2021-02-11T00:18:42.926 INFO:root:teuthology version: 1.1.0-14fad317 2021-02-11T00:18:42.927 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} --verbose --name aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi --archive /home/teuthworker/archive/aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi/5874431 --owner scheduled_aemerson@teuthology -- /home/teuthworker/archive/aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi/5874431/orig.config.yaml 2021-02-11T00:18:42.966 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-02-11T00:18:43.028 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi/5874431 branch: wip-datalog-genbacking description: rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: aemerson@redhat.com first_in_suite: false job_id: '5874431' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: wip-datalog-genbacking ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 597c1051df9f309aa7d6c6db46dcbc2c3982c6e3 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 597c1051df9f309aa7d6c6db46dcbc2c3982c6e3 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: wip-datalog-genbacking sha1: 597c1051df9f309aa7d6c6db46dcbc2c3982c6e3 owner: scheduled_aemerson@teuthology priority: 68 repo: git://git.ceph.com/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 597c1051df9f309aa7d6c6db46dcbc2c3982c6e3 sleep_before_teardown: 0 suite: rgw suite_branch: wip-datalog-genbacking suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 597c1051df9f309aa7d6c6db46dcbc2c3982c6e3 targets: smithi057.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKoCvpwFe8apsck2MdrWK3EDZV8EosBNVkB/RSE/yQxHn50Ykjz3HAw8V3iKist9ypyMYwUH6KBS+Nn0x4zE6rbr0psdkO5PoNVvOozmdswDNzLbfFuT3l34PG/m0V/LMMWkTBNdvnxP66WjA8kScX5YLa3TFxCReuQAW8CNPEipAzHVgW3NQhD+M/xR4kwV94O9O/QVEiQsblbbrsas74gf/U58cU8nCWkDiymUUhxl23YvtuxjEfRslH3zoC0L5IMJiW92hCLxccM8HX1ZlO7ZLBiaor4JXRayfSFYwND85bXct1kJ/zjbgIE5ZgnB7FoQHtSZnKeN56W5XOD4Xf smithi095.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj65BMVOnEAISCOlnW3qjVHQt+aIJPq3o/p+le9aqXL2+AQ8VXsBSewIDMWDtIXWmMa//mBFRa9x0oGKXHWPK7w+n23MUBlwQHnTHvPGpczSjnb+PXmwBIZYaTRl7gyl6ts8aecf/FrTM54fhGyMMGYXh3ZIJwuh9NyBMf9BKnRfsBQ6jTkbTKbMKL7XmRisu3EcaelLPM3aiLpuyCYw66OgBdMPvC4Bgvo9+J1lRVdivlw3PuagJn4soMxyAFF6zXJ6DnhAX0PwEsJmK6PrQrBwZ7bFy+lXFq8/bDsGwPrG8cynvH8kMKCjl/VoP1wPKwbg3e8XK/Ci2SQ48VPvMR tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master timestamp: 2021-02-11_00:10:51 tube: smithi user: aemerson verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.29888 2021-02-11T00:18:43.028 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa; will attempt to use it 2021-02-11T00:18:43.037 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks 2021-02-11T00:18:43.038 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-02-11T00:18:43.039 INFO:teuthology.task.internal:Checking packages... 2021-02-11T00:18:43.093 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '597c1051df9f309aa7d6c6db46dcbc2c3982c6e3' 2021-02-11T00:18:43.093 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-02-11T00:18:43.093 INFO:teuthology.packaging:ref: None 2021-02-11T00:18:43.093 INFO:teuthology.packaging:tag: None 2021-02-11T00:18:43.094 INFO:teuthology.packaging:branch: wip-datalog-genbacking 2021-02-11T00:18:43.094 INFO:teuthology.packaging:sha1: 597c1051df9f309aa7d6c6db46dcbc2c3982c6e3 2021-02-11T00:18:43.094 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-datalog-genbacking 2021-02-11T00:18:43.288 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-691-g597c1051-1bionic 2021-02-11T00:18:43.288 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-02-11T00:18:43.295 INFO:teuthology.task.internal:no buildpackages task found 2021-02-11T00:18:43.295 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-02-11T00:18:44.526 INFO:teuthology.task.internal:Saving configuration 2021-02-11T00:18:44.540 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-02-11T00:18:45.954 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-11T00:18:45.972 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-11 00:13:22.709151', 'description': '/home/teuthworker/archive/aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi/5874431', 'locked_by': 'scheduled_aemerson@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM5SA6sVz/ZGPHpCt729u0B3ONnlO2lVR48X5RSjVH2I', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi057.front.sepia.ceph.com'} 2021-02-11T00:18:45.989 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-11 00:13:22.709130', 'description': '/home/teuthworker/archive/aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi/5874431', 'locked_by': 'scheduled_aemerson@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII2xC2vfsvkMtjWdlDZ6O7Q4I7HSAtF6Lll/u63yyZJt', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi095.front.sepia.ceph.com'} 2021-02-11T00:18:45.990 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-02-11T00:18:46.877 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2021-02-11T00:18:46.877 INFO:teuthology.task.internal:roles: ubuntu@smithi095.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2021-02-11T00:18:46.877 INFO:teuthology.run_tasks:Running task console_log... 2021-02-11T00:18:47.988 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fea891d92f0>, signals=[15]) 2021-02-11T00:18:47.989 INFO:teuthology.run_tasks:Running task internal.connect... 2021-02-11T00:18:49.040 INFO:teuthology.task.internal:Opening connections... 2021-02-11T00:18:49.041 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi057.front.sepia.ceph.com 2021-02-11T00:18:49.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T00:18:49.116 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi095.front.sepia.ceph.com 2021-02-11T00:18:49.118 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi095.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T00:18:49.191 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-02-11T00:18:50.337 DEBUG:teuthology.orchestra.run.smithi057:> uname -m 2021-02-11T00:18:50.344 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64 2021-02-11T00:18:50.345 DEBUG:teuthology.orchestra.run.smithi057:> cat /etc/os-release 2021-02-11T00:18:50.390 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="Ubuntu" 2021-02-11T00:18:50.390 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-11T00:18:50.391 INFO:teuthology.orchestra.run.smithi057.stdout:ID=ubuntu 2021-02-11T00:18:50.391 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE=debian 2021-02-11T00:18:50.391 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-11T00:18:50.391 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="18.04" 2021-02-11T00:18:50.391 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-11T00:18:50.391 INFO:teuthology.orchestra.run.smithi057.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-11T00:18:50.391 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-11T00:18:50.391 INFO:teuthology.orchestra.run.smithi057.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-11T00:18:50.391 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_CODENAME=bionic 2021-02-11T00:18:50.391 INFO:teuthology.orchestra.run.smithi057.stdout:UBUNTU_CODENAME=bionic 2021-02-11T00:18:50.392 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2021-02-11T00:18:50.499 DEBUG:teuthology.orchestra.run.smithi095:> uname -m 2021-02-11T00:18:50.505 INFO:teuthology.orchestra.run.smithi095.stdout:x86_64 2021-02-11T00:18:50.506 DEBUG:teuthology.orchestra.run.smithi095:> cat /etc/os-release 2021-02-11T00:18:50.553 INFO:teuthology.orchestra.run.smithi095.stdout:NAME="Ubuntu" 2021-02-11T00:18:50.553 INFO:teuthology.orchestra.run.smithi095.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-11T00:18:50.553 INFO:teuthology.orchestra.run.smithi095.stdout:ID=ubuntu 2021-02-11T00:18:50.553 INFO:teuthology.orchestra.run.smithi095.stdout:ID_LIKE=debian 2021-02-11T00:18:50.553 INFO:teuthology.orchestra.run.smithi095.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-11T00:18:50.554 INFO:teuthology.orchestra.run.smithi095.stdout:VERSION_ID="18.04" 2021-02-11T00:18:50.554 INFO:teuthology.orchestra.run.smithi095.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-11T00:18:50.554 INFO:teuthology.orchestra.run.smithi095.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-11T00:18:50.554 INFO:teuthology.orchestra.run.smithi095.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-11T00:18:50.554 INFO:teuthology.orchestra.run.smithi095.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-11T00:18:50.554 INFO:teuthology.orchestra.run.smithi095.stdout:VERSION_CODENAME=bionic 2021-02-11T00:18:50.554 INFO:teuthology.orchestra.run.smithi095.stdout:UBUNTU_CODENAME=bionic 2021-02-11T00:18:50.554 INFO:teuthology.lock.ops:Updating smithi095.front.sepia.ceph.com on lock server 2021-02-11T00:18:50.612 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-02-11T00:18:51.182 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-02-11T00:18:52.916 INFO:teuthology.task.internal:Checking for old test directory... 2021-02-11T00:18:52.916 DEBUG:teuthology.orchestra.run.smithi057:> test '!' -e /home/ubuntu/cephtest 2021-02-11T00:18:52.918 DEBUG:teuthology.orchestra.run.smithi095:> test '!' -e /home/ubuntu/cephtest 2021-02-11T00:18:52.924 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-02-11T00:18:55.251 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-02-11T00:18:55.251 DEBUG:teuthology.orchestra.run.smithi057:> test -z $(ls -A /var/lib/ceph) 2021-02-11T00:18:55.254 DEBUG:teuthology.orchestra.run.smithi095:> test -z $(ls -A /var/lib/ceph) 2021-02-11T00:18:55.272 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-02-11T00:18:57.022 INFO:teuthology.run_tasks:Running task kernel... 2021-02-11T00:18:58.195 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2021-02-11T00:18:58.195 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-02-11T00:18:58.196 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2021-02-11T00:18:58.203 INFO:teuthology.orchestra.run.smithi057.stdout:4.15.0-134-generic 2021-02-11T00:18:58.203 INFO:teuthology.task.kernel:Running kernel on smithi057: 4.15.0-134-generic 2021-02-11T00:18:58.204 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt-get clean 2021-02-11T00:18:58.312 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt-get update 2021-02-11T00:18:58.564 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T00:18:58.578 INFO:teuthology.orchestra.run.smithi057.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-02-11T00:18:58.631 INFO:teuthology.orchestra.run.smithi057.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-02-11T00:18:58.945 INFO:teuthology.orchestra.run.smithi057.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-02-11T00:18:58.996 INFO:teuthology.orchestra.run.smithi057.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [915 kB] 2021-02-11T00:18:59.122 INFO:teuthology.orchestra.run.smithi057.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,219 kB] 2021-02-11T00:18:59.395 INFO:teuthology.orchestra.run.smithi057.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,543 kB] 2021-02-11T00:18:59.503 INFO:teuthology.orchestra.run.smithi057.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [298 kB] 2021-02-11T00:18:59.508 INFO:teuthology.orchestra.run.smithi057.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,883 kB] 2021-02-11T00:18:59.518 INFO:teuthology.orchestra.run.smithi057.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [226 kB] 2021-02-11T00:18:59.528 INFO:teuthology.orchestra.run.smithi057.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [29.9 kB] 2021-02-11T00:18:59.529 INFO:teuthology.orchestra.run.smithi057.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [975 kB] 2021-02-11T00:18:59.570 INFO:teuthology.orchestra.run.smithi057.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,109 kB] 2021-02-11T00:18:59.600 INFO:teuthology.orchestra.run.smithi057.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [248 kB] 2021-02-11T00:18:59.612 INFO:teuthology.orchestra.run.smithi057.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [390 kB] 2021-02-11T00:18:59.629 INFO:teuthology.orchestra.run.smithi057.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [247 kB] 2021-02-11T00:18:59.640 INFO:teuthology.orchestra.run.smithi057.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [33.4 kB] 2021-02-11T00:18:59.641 INFO:teuthology.orchestra.run.smithi057.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,718 kB] 2021-02-11T00:18:59.693 INFO:teuthology.orchestra.run.smithi057.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,560 kB] 2021-02-11T00:18:59.735 INFO:teuthology.orchestra.run.smithi057.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [363 kB] 2021-02-11T00:18:59.747 INFO:teuthology.orchestra.run.smithi057.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [14.3 kB] 2021-02-11T00:18:59.747 INFO:teuthology.orchestra.run.smithi057.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [31.8 kB] 2021-02-11T00:18:59.747 INFO:teuthology.orchestra.run.smithi057.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,300 B] 2021-02-11T00:19:01.160 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 13.1 MB in 3s (4,982 kB/s) 2021-02-11T00:19:02.628 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2021-02-11T00:19:02.678 DEBUG:teuthology.orchestra.run.smithi057:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-11T00:19:02.760 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2021-02-11T00:19:03.011 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2021-02-11T00:19:03.011 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2021-02-11T00:19:03.101 INFO:teuthology.orchestra.run.smithi057.stderr:E: Unable to locate package linux-image-current-generic 2021-02-11T00:19:03.102 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-11T00:19:03.103 DEBUG:teuthology.orchestra.run.smithi057:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-11T00:19:03.185 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2021-02-11T00:19:03.422 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2021-02-11T00:19:03.423 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2021-02-11T00:19:03.605 INFO:teuthology.orchestra.run.smithi057.stdout:The following additional packages will be installed: 2021-02-11T00:19:03.605 INFO:teuthology.orchestra.run.smithi057.stdout: linux-generic linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-11T00:19:03.606 INFO:teuthology.orchestra.run.smithi057.stdout: linux-headers-generic linux-image-4.15.0-135-generic 2021-02-11T00:19:03.606 INFO:teuthology.orchestra.run.smithi057.stdout: linux-modules-4.15.0-135-generic linux-modules-extra-4.15.0-135-generic 2021-02-11T00:19:03.607 INFO:teuthology.orchestra.run.smithi057.stdout:Suggested packages: 2021-02-11T00:19:03.607 INFO:teuthology.orchestra.run.smithi057.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2021-02-11T00:19:03.607 INFO:teuthology.orchestra.run.smithi057.stdout:Recommended packages: 2021-02-11T00:19:03.608 INFO:teuthology.orchestra.run.smithi057.stdout: thermald 2021-02-11T00:19:03.641 INFO:teuthology.orchestra.run.smithi057.stdout:The following NEW packages will be installed: 2021-02-11T00:19:03.642 INFO:teuthology.orchestra.run.smithi057.stdout: linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-11T00:19:03.642 INFO:teuthology.orchestra.run.smithi057.stdout: linux-image-4.15.0-135-generic linux-modules-4.15.0-135-generic 2021-02-11T00:19:03.642 INFO:teuthology.orchestra.run.smithi057.stdout: linux-modules-extra-4.15.0-135-generic 2021-02-11T00:19:03.644 INFO:teuthology.orchestra.run.smithi057.stdout:The following packages will be upgraded: 2021-02-11T00:19:03.645 INFO:teuthology.orchestra.run.smithi057.stdout: linux-generic linux-headers-generic linux-image-generic 2021-02-11T00:19:03.820 INFO:teuthology.orchestra.run.smithi057.stdout:3 upgraded, 5 newly installed, 0 to remove and 23 not upgraded. 2021-02-11T00:19:03.820 INFO:teuthology.orchestra.run.smithi057.stdout:Need to get 66.0 MB of archives. 2021-02-11T00:19:03.820 INFO:teuthology.orchestra.run.smithi057.stdout:After this operation, 334 MB of additional disk space will be used. 2021-02-11T00:19:03.820 INFO:teuthology.orchestra.run.smithi057.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-135-generic amd64 4.15.0-135.139 [13.1 MB] 2021-02-11T00:19:04.905 INFO:teuthology.orchestra.run.smithi057.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-135-generic amd64 4.15.0-135.139 [8,019 kB] 2021-02-11T00:19:05.147 INFO:teuthology.orchestra.run.smithi057.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-135-generic amd64 4.15.0-135.139 [32.8 MB] 2021-02-11T00:19:06.038 INFO:teuthology.orchestra.run.smithi057.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.135.122 [1,868 B] 2021-02-11T00:19:06.038 INFO:teuthology.orchestra.run.smithi057.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.135.122 [2,592 B] 2021-02-11T00:19:06.038 INFO:teuthology.orchestra.run.smithi057.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135 all 4.15.0-135.139 [10.9 MB] 2021-02-11T00:19:06.341 INFO:teuthology.orchestra.run.smithi057.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135-generic amd64 4.15.0-135.139 [1,097 kB] 2021-02-11T00:19:06.362 INFO:teuthology.orchestra.run.smithi057.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.135.122 [2,488 B] 2021-02-11T00:19:07.009 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 66.0 MB in 3s (24.4 MB/s) 2021-02-11T00:19:07.155 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package linux-modules-4.15.0-135-generic. 2021-02-11T00:19:07.197 INFO:teuthology.orchestra.run.smithi057.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102079 files and directories currently installed.) 2021-02-11T00:19:07.200 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../0-linux-modules-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T00:19:07.244 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:19:08.862 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package linux-image-4.15.0-135-generic. 2021-02-11T00:19:08.874 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../1-linux-image-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T00:19:08.965 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:19:09.691 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-135-generic. 2021-02-11T00:19:09.704 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T00:19:09.741 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:19:13.321 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../3-linux-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T00:19:13.440 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking linux-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T00:19:13.772 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T00:19:13.893 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking linux-image-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T00:19:14.238 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package linux-headers-4.15.0-135. 2021-02-11T00:19:14.251 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../5-linux-headers-4.15.0-135_4.15.0-135.139_all.deb ... 2021-02-11T00:19:14.288 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-11T00:19:19.035 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package linux-headers-4.15.0-135-generic. 2021-02-11T00:19:19.057 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../6-linux-headers-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T00:19:19.101 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:19:21.778 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T00:19:21.900 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking linux-headers-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T00:19:22.233 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:19:22.804 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-11T00:19:22.939 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:19:23.374 INFO:teuthology.orchestra.run.smithi057.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-135-generic 2021-02-11T00:19:23.374 INFO:teuthology.orchestra.run.smithi057.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-135-generic 2021-02-11T00:19:23.491 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:19:23.992 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:19:24.118 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up linux-image-generic (4.15.0.135.122) ... 2021-02-11T00:19:24.386 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up linux-headers-generic (4.15.0.135.122) ... 2021-02-11T00:19:24.512 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up linux-generic (4.15.0.135.122) ... 2021-02-11T00:19:24.638 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:19:24.838 INFO:teuthology.orchestra.run.smithi057.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-02-11T00:19:24.838 INFO:teuthology.orchestra.run.smithi057.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-135-generic 2021-02-11T00:19:40.213 INFO:teuthology.orchestra.run.smithi057.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-02-11T00:19:40.213 INFO:teuthology.orchestra.run.smithi057.stdout:Sourcing file `/etc/default/grub' 2021-02-11T00:19:40.251 INFO:teuthology.orchestra.run.smithi057.stdout:Generating grub configuration file ... 2021-02-11T00:19:40.552 INFO:teuthology.orchestra.run.smithi057.stdout:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-11T00:19:40.586 INFO:teuthology.orchestra.run.smithi057.stdout:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-11T00:19:40.902 INFO:teuthology.orchestra.run.smithi057.stdout:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-11T00:19:40.931 INFO:teuthology.orchestra.run.smithi057.stdout:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-11T00:19:41.799 INFO:teuthology.orchestra.run.smithi057.stdout:done 2021-02-11T00:19:42.195 DEBUG:teuthology.orchestra.run.smithi057:> dpkg -s linux-image-generic 2021-02-11T00:19:42.217 INFO:teuthology.orchestra.run.smithi057.stdout:Package: linux-image-generic 2021-02-11T00:19:42.217 INFO:teuthology.orchestra.run.smithi057.stdout:Status: install ok installed 2021-02-11T00:19:42.217 INFO:teuthology.orchestra.run.smithi057.stdout:Priority: optional 2021-02-11T00:19:42.218 INFO:teuthology.orchestra.run.smithi057.stdout:Section: kernel 2021-02-11T00:19:42.218 INFO:teuthology.orchestra.run.smithi057.stdout:Installed-Size: 17 2021-02-11T00:19:42.218 INFO:teuthology.orchestra.run.smithi057.stdout:Maintainer: Ubuntu Kernel Team 2021-02-11T00:19:42.218 INFO:teuthology.orchestra.run.smithi057.stdout:Architecture: amd64 2021-02-11T00:19:42.218 INFO:teuthology.orchestra.run.smithi057.stdout:Source: linux-meta 2021-02-11T00:19:42.218 INFO:teuthology.orchestra.run.smithi057.stdout:Version: 4.15.0.135.122 2021-02-11T00:19:42.218 INFO:teuthology.orchestra.run.smithi057.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-11T00:19:42.218 INFO:teuthology.orchestra.run.smithi057.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-11T00:19:42.218 INFO:teuthology.orchestra.run.smithi057.stdout:Recommends: thermald 2021-02-11T00:19:42.218 INFO:teuthology.orchestra.run.smithi057.stdout:Description: Generic Linux kernel image 2021-02-11T00:19:42.218 INFO:teuthology.orchestra.run.smithi057.stdout: This package will always depend on the latest generic kernel image 2021-02-11T00:19:42.218 INFO:teuthology.orchestra.run.smithi057.stdout: available. 2021-02-11T00:19:42.219 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-134-generic Expected: 4.15.0-135-generic 2021-02-11T00:19:42.219 DEBUG:teuthology.orchestra.run.smithi095:> uname -r 2021-02-11T00:19:42.225 INFO:teuthology.orchestra.run.smithi095.stdout:4.15.0-134-generic 2021-02-11T00:19:42.225 INFO:teuthology.task.kernel:Running kernel on smithi095: 4.15.0-134-generic 2021-02-11T00:19:42.226 DEBUG:teuthology.orchestra.run.smithi095:> sudo apt-get clean 2021-02-11T00:19:42.336 DEBUG:teuthology.orchestra.run.smithi095:> sudo apt-get update 2021-02-11T00:19:42.614 INFO:teuthology.orchestra.run.smithi095.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T00:19:42.615 INFO:teuthology.orchestra.run.smithi095.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-02-11T00:19:42.663 INFO:teuthology.orchestra.run.smithi095.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-02-11T00:19:42.911 INFO:teuthology.orchestra.run.smithi095.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-02-11T00:19:43.015 INFO:teuthology.orchestra.run.smithi095.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,543 kB] 2021-02-11T00:19:43.145 INFO:teuthology.orchestra.run.smithi095.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,883 kB] 2021-02-11T00:19:43.445 INFO:teuthology.orchestra.run.smithi095.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [915 kB] 2021-02-11T00:19:43.464 INFO:teuthology.orchestra.run.smithi095.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [298 kB] 2021-02-11T00:19:43.469 INFO:teuthology.orchestra.run.smithi095.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [226 kB] 2021-02-11T00:19:43.473 INFO:teuthology.orchestra.run.smithi095.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [29.9 kB] 2021-02-11T00:19:43.473 INFO:teuthology.orchestra.run.smithi095.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,109 kB] 2021-02-11T00:19:43.530 INFO:teuthology.orchestra.run.smithi095.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,219 kB] 2021-02-11T00:19:43.542 INFO:teuthology.orchestra.run.smithi095.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [975 kB] 2021-02-11T00:19:43.565 INFO:teuthology.orchestra.run.smithi095.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [390 kB] 2021-02-11T00:19:43.566 INFO:teuthology.orchestra.run.smithi095.stdout:Get:15 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [248 kB] 2021-02-11T00:19:43.576 INFO:teuthology.orchestra.run.smithi095.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [247 kB] 2021-02-11T00:19:43.584 INFO:teuthology.orchestra.run.smithi095.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [33.4 kB] 2021-02-11T00:19:43.584 INFO:teuthology.orchestra.run.smithi095.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,718 kB] 2021-02-11T00:19:43.653 INFO:teuthology.orchestra.run.smithi095.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,560 kB] 2021-02-11T00:19:43.697 INFO:teuthology.orchestra.run.smithi095.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [363 kB] 2021-02-11T00:19:43.701 INFO:teuthology.orchestra.run.smithi095.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [14.3 kB] 2021-02-11T00:19:43.701 INFO:teuthology.orchestra.run.smithi095.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [31.8 kB] 2021-02-11T00:19:43.701 INFO:teuthology.orchestra.run.smithi095.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,300 B] 2021-02-11T00:19:45.126 INFO:teuthology.orchestra.run.smithi095.stdout:Fetched 13.1 MB in 3s (5,089 kB/s) 2021-02-11T00:19:46.535 INFO:teuthology.orchestra.run.smithi095.stdout:Reading package lists... 2021-02-11T00:19:46.581 DEBUG:teuthology.orchestra.run.smithi095:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-11T00:19:46.657 INFO:teuthology.orchestra.run.smithi095.stdout:Reading package lists... 2021-02-11T00:19:46.885 INFO:teuthology.orchestra.run.smithi095.stdout:Building dependency tree... 2021-02-11T00:19:46.886 INFO:teuthology.orchestra.run.smithi095.stdout:Reading state information... 2021-02-11T00:19:46.965 INFO:teuthology.orchestra.run.smithi095.stderr:E: Unable to locate package linux-image-current-generic 2021-02-11T00:19:46.966 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-11T00:19:46.967 DEBUG:teuthology.orchestra.run.smithi095:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-11T00:19:47.049 INFO:teuthology.orchestra.run.smithi095.stdout:Reading package lists... 2021-02-11T00:19:47.293 INFO:teuthology.orchestra.run.smithi095.stdout:Building dependency tree... 2021-02-11T00:19:47.293 INFO:teuthology.orchestra.run.smithi095.stdout:Reading state information... 2021-02-11T00:19:47.469 INFO:teuthology.orchestra.run.smithi095.stdout:The following additional packages will be installed: 2021-02-11T00:19:47.470 INFO:teuthology.orchestra.run.smithi095.stdout: linux-generic linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-11T00:19:47.470 INFO:teuthology.orchestra.run.smithi095.stdout: linux-headers-generic linux-image-4.15.0-135-generic 2021-02-11T00:19:47.470 INFO:teuthology.orchestra.run.smithi095.stdout: linux-modules-4.15.0-135-generic linux-modules-extra-4.15.0-135-generic 2021-02-11T00:19:47.471 INFO:teuthology.orchestra.run.smithi095.stdout:Suggested packages: 2021-02-11T00:19:47.472 INFO:teuthology.orchestra.run.smithi095.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2021-02-11T00:19:47.472 INFO:teuthology.orchestra.run.smithi095.stdout:Recommended packages: 2021-02-11T00:19:47.472 INFO:teuthology.orchestra.run.smithi095.stdout: thermald 2021-02-11T00:19:47.503 INFO:teuthology.orchestra.run.smithi095.stdout:The following NEW packages will be installed: 2021-02-11T00:19:47.503 INFO:teuthology.orchestra.run.smithi095.stdout: linux-headers-4.15.0-135 linux-headers-4.15.0-135-generic 2021-02-11T00:19:47.504 INFO:teuthology.orchestra.run.smithi095.stdout: linux-image-4.15.0-135-generic linux-modules-4.15.0-135-generic 2021-02-11T00:19:47.504 INFO:teuthology.orchestra.run.smithi095.stdout: linux-modules-extra-4.15.0-135-generic 2021-02-11T00:19:47.505 INFO:teuthology.orchestra.run.smithi095.stdout:The following packages will be upgraded: 2021-02-11T00:19:47.506 INFO:teuthology.orchestra.run.smithi095.stdout: linux-generic linux-headers-generic linux-image-generic 2021-02-11T00:19:47.683 INFO:teuthology.orchestra.run.smithi095.stdout:3 upgraded, 5 newly installed, 0 to remove and 23 not upgraded. 2021-02-11T00:19:47.683 INFO:teuthology.orchestra.run.smithi095.stdout:Need to get 66.0 MB of archives. 2021-02-11T00:19:47.683 INFO:teuthology.orchestra.run.smithi095.stdout:After this operation, 334 MB of additional disk space will be used. 2021-02-11T00:19:47.683 INFO:teuthology.orchestra.run.smithi095.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-135-generic amd64 4.15.0-135.139 [13.1 MB] 2021-02-11T00:19:48.692 INFO:teuthology.orchestra.run.smithi095.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-135-generic amd64 4.15.0-135.139 [8,019 kB] 2021-02-11T00:19:48.893 INFO:teuthology.orchestra.run.smithi095.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-135-generic amd64 4.15.0-135.139 [32.8 MB] 2021-02-11T00:19:49.884 INFO:teuthology.orchestra.run.smithi095.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.135.122 [1,868 B] 2021-02-11T00:19:49.884 INFO:teuthology.orchestra.run.smithi095.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.135.122 [2,592 B] 2021-02-11T00:19:49.884 INFO:teuthology.orchestra.run.smithi095.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135 all 4.15.0-135.139 [10.9 MB] 2021-02-11T00:19:50.324 INFO:teuthology.orchestra.run.smithi095.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-135-generic amd64 4.15.0-135.139 [1,097 kB] 2021-02-11T00:19:50.353 INFO:teuthology.orchestra.run.smithi095.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.135.122 [2,488 B] 2021-02-11T00:19:50.830 INFO:teuthology.orchestra.run.smithi095.stdout:Fetched 66.0 MB in 3s (23.3 MB/s) 2021-02-11T00:19:50.996 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package linux-modules-4.15.0-135-generic. 2021-02-11T00:19:51.032 INFO:teuthology.orchestra.run.smithi095.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102079 files and directories currently installed.) 2021-02-11T00:19:51.035 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../0-linux-modules-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T00:19:51.085 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:19:52.685 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package linux-image-4.15.0-135-generic. 2021-02-11T00:19:52.696 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../1-linux-image-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T00:19:52.792 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:19:53.272 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-135-generic. 2021-02-11T00:19:53.285 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T00:19:53.322 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:19:57.039 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../3-linux-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T00:19:57.160 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking linux-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T00:19:57.468 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T00:19:57.597 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking linux-image-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T00:19:57.933 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package linux-headers-4.15.0-135. 2021-02-11T00:19:57.946 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../5-linux-headers-4.15.0-135_4.15.0-135.139_all.deb ... 2021-02-11T00:19:57.983 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-11T00:20:02.752 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package linux-headers-4.15.0-135-generic. 2021-02-11T00:20:02.774 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../6-linux-headers-4.15.0-135-generic_4.15.0-135.139_amd64.deb ... 2021-02-11T00:20:02.818 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:20:05.241 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.135.122_amd64.deb ... 2021-02-11T00:20:05.365 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking linux-headers-generic (4.15.0.135.122) over (4.15.0.134.121) ... 2021-02-11T00:20:05.684 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up linux-modules-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:20:06.214 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up linux-headers-4.15.0-135 (4.15.0-135.139) ... 2021-02-11T00:20:06.357 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:20:06.786 INFO:teuthology.orchestra.run.smithi095.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-135-generic 2021-02-11T00:20:06.786 INFO:teuthology.orchestra.run.smithi095.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-135-generic 2021-02-11T00:20:06.925 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up linux-modules-extra-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:20:07.410 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up linux-headers-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:20:07.536 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up linux-image-generic (4.15.0.135.122) ... 2021-02-11T00:20:07.637 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up linux-headers-generic (4.15.0.135.122) ... 2021-02-11T00:20:07.788 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up linux-generic (4.15.0.135.122) ... 2021-02-11T00:20:07.923 INFO:teuthology.orchestra.run.smithi095.stdout:Processing triggers for linux-image-4.15.0-135-generic (4.15.0-135.139) ... 2021-02-11T00:20:08.131 INFO:teuthology.orchestra.run.smithi095.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-02-11T00:20:08.131 INFO:teuthology.orchestra.run.smithi095.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-135-generic 2021-02-11T00:20:22.436 INFO:teuthology.orchestra.run.smithi095.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-02-11T00:20:22.436 INFO:teuthology.orchestra.run.smithi095.stdout:Sourcing file `/etc/default/grub' 2021-02-11T00:20:22.472 INFO:teuthology.orchestra.run.smithi095.stdout:Generating grub configuration file ... 2021-02-11T00:20:22.885 INFO:teuthology.orchestra.run.smithi095.stdout:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-11T00:20:22.916 INFO:teuthology.orchestra.run.smithi095.stdout:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-11T00:20:23.298 INFO:teuthology.orchestra.run.smithi095.stdout:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-11T00:20:23.334 INFO:teuthology.orchestra.run.smithi095.stdout:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-11T00:20:24.313 INFO:teuthology.orchestra.run.smithi095.stdout:done 2021-02-11T00:20:24.731 DEBUG:teuthology.orchestra.run.smithi095:> dpkg -s linux-image-generic 2021-02-11T00:20:24.751 INFO:teuthology.orchestra.run.smithi095.stdout:Package: linux-image-generic 2021-02-11T00:20:24.751 INFO:teuthology.orchestra.run.smithi095.stdout:Status: install ok installed 2021-02-11T00:20:24.752 INFO:teuthology.orchestra.run.smithi095.stdout:Priority: optional 2021-02-11T00:20:24.752 INFO:teuthology.orchestra.run.smithi095.stdout:Section: kernel 2021-02-11T00:20:24.752 INFO:teuthology.orchestra.run.smithi095.stdout:Installed-Size: 17 2021-02-11T00:20:24.752 INFO:teuthology.orchestra.run.smithi095.stdout:Maintainer: Ubuntu Kernel Team 2021-02-11T00:20:24.752 INFO:teuthology.orchestra.run.smithi095.stdout:Architecture: amd64 2021-02-11T00:20:24.752 INFO:teuthology.orchestra.run.smithi095.stdout:Source: linux-meta 2021-02-11T00:20:24.752 INFO:teuthology.orchestra.run.smithi095.stdout:Version: 4.15.0.135.122 2021-02-11T00:20:24.752 INFO:teuthology.orchestra.run.smithi095.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-11T00:20:24.752 INFO:teuthology.orchestra.run.smithi095.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-11T00:20:24.752 INFO:teuthology.orchestra.run.smithi095.stdout:Recommends: thermald 2021-02-11T00:20:24.752 INFO:teuthology.orchestra.run.smithi095.stdout:Description: Generic Linux kernel image 2021-02-11T00:20:24.752 INFO:teuthology.orchestra.run.smithi095.stdout: This package will always depend on the latest generic kernel image 2021-02-11T00:20:24.752 INFO:teuthology.orchestra.run.smithi095.stdout: available. 2021-02-11T00:20:24.753 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-134-generic Expected: 4.15.0-135-generic 2021-02-11T00:20:24.753 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-02-11T00:20:24.753 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-02-11T00:20:24.753 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-02-11T00:20:24.753 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2021-02-11T00:20:24.753 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi057.front.sepia.ceph.com, path=None, version=distro) 2021-02-11T00:20:24.754 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt-get clean 2021-02-11T00:20:24.811 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt-get update 2021-02-11T00:20:24.946 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-11T00:20:25.051 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T00:20:25.131 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-11T00:20:25.210 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-11T00:20:26.884 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2021-02-11T00:20:26.935 DEBUG:teuthology.orchestra.run.smithi057:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-11T00:20:27.013 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2021-02-11T00:20:27.258 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2021-02-11T00:20:27.258 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2021-02-11T00:20:27.344 INFO:teuthology.orchestra.run.smithi057.stderr:E: Unable to locate package linux-image-current-generic 2021-02-11T00:20:27.345 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-11T00:20:27.345 DEBUG:teuthology.orchestra.run.smithi057:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-11T00:20:27.426 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2021-02-11T00:20:27.665 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2021-02-11T00:20:27.666 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2021-02-11T00:20:27.877 INFO:teuthology.orchestra.run.smithi057.stdout:linux-image-generic is already the newest version (4.15.0.135.122). 2021-02-11T00:20:27.877 INFO:teuthology.orchestra.run.smithi057.stdout:0 upgraded, 0 newly installed, 0 to remove and 23 not upgraded. 2021-02-11T00:20:27.879 DEBUG:teuthology.orchestra.run.smithi057:> dpkg -s linux-image-generic 2021-02-11T00:20:27.900 INFO:teuthology.orchestra.run.smithi057.stdout:Package: linux-image-generic 2021-02-11T00:20:27.901 INFO:teuthology.orchestra.run.smithi057.stdout:Status: install ok installed 2021-02-11T00:20:27.901 INFO:teuthology.orchestra.run.smithi057.stdout:Priority: optional 2021-02-11T00:20:27.901 INFO:teuthology.orchestra.run.smithi057.stdout:Section: kernel 2021-02-11T00:20:27.901 INFO:teuthology.orchestra.run.smithi057.stdout:Installed-Size: 17 2021-02-11T00:20:27.901 INFO:teuthology.orchestra.run.smithi057.stdout:Maintainer: Ubuntu Kernel Team 2021-02-11T00:20:27.901 INFO:teuthology.orchestra.run.smithi057.stdout:Architecture: amd64 2021-02-11T00:20:27.901 INFO:teuthology.orchestra.run.smithi057.stdout:Source: linux-meta 2021-02-11T00:20:27.901 INFO:teuthology.orchestra.run.smithi057.stdout:Version: 4.15.0.135.122 2021-02-11T00:20:27.901 INFO:teuthology.orchestra.run.smithi057.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-11T00:20:27.901 INFO:teuthology.orchestra.run.smithi057.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-11T00:20:27.901 INFO:teuthology.orchestra.run.smithi057.stdout:Recommends: thermald 2021-02-11T00:20:27.901 INFO:teuthology.orchestra.run.smithi057.stdout:Description: Generic Linux kernel image 2021-02-11T00:20:27.902 INFO:teuthology.orchestra.run.smithi057.stdout: This package will always depend on the latest generic kernel image 2021-02-11T00:20:27.902 INFO:teuthology.orchestra.run.smithi057.stdout: available. 2021-02-11T00:20:27.902 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2021-02-11T00:20:27.948 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.oV4CVlgV7L 2021-02-11T00:20:27.948 DEBUG:teuthology.orchestra.run.smithi057:> sudo cp /boot/grub/grub.cfg /tmp/tmp.oV4CVlgV7L 2021-02-11T00:20:28.010 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.oV4CVlgV7L 2021-02-11T00:20:28.126 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.oV4CVlgV7L is 8KB 2021-02-11T00:20:28.138 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.oV4CVlgV7L 2021-02-11T00:20:28.145 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-02-11T00:20:28.203 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:20:28.203 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-02-11T00:20:28.203 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-02-11T00:20:28.283 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-135-generic 2021-02-11T00:20:28.284 DEBUG:teuthology.orchestra.run.smithi057:> sudo update-grub 2021-02-11T00:20:29.565 INFO:teuthology.orchestra.run.smithi057.stderr:Sourcing file `/etc/default/grub' 2021-02-11T00:20:29.578 INFO:teuthology.orchestra.run.smithi057.stderr:Generating grub configuration file ... 2021-02-11T00:20:29.806 INFO:teuthology.orchestra.run.smithi057.stderr:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-11T00:20:29.817 INFO:teuthology.orchestra.run.smithi057.stderr:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-11T00:20:30.162 INFO:teuthology.orchestra.run.smithi057.stderr:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-11T00:20:30.175 INFO:teuthology.orchestra.run.smithi057.stderr:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-11T00:20:30.791 INFO:teuthology.orchestra.run.smithi057.stderr:done 2021-02-11T00:20:30.970 DEBUG:teuthology.orchestra.run.smithi057:> sudo shutdown -r now 2021-02-11T00:20:30.972 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2021-02-11T00:20:30.972 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi095.front.sepia.ceph.com, path=None, version=distro) 2021-02-11T00:20:30.972 DEBUG:teuthology.orchestra.run.smithi095:> sudo apt-get clean 2021-02-11T00:20:31.028 DEBUG:teuthology.orchestra.run.smithi095:> sudo apt-get update 2021-02-11T00:20:31.157 INFO:teuthology.orchestra.run.smithi095.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-02-11T00:20:31.264 INFO:teuthology.orchestra.run.smithi095.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T00:20:31.343 INFO:teuthology.orchestra.run.smithi095.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-02-11T00:20:31.422 INFO:teuthology.orchestra.run.smithi095.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-02-11T00:20:33.025 INFO:teuthology.orchestra.run.smithi095.stdout:Reading package lists... 2021-02-11T00:20:33.073 DEBUG:teuthology.orchestra.run.smithi095:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-02-11T00:20:33.149 INFO:teuthology.orchestra.run.smithi095.stdout:Reading package lists... 2021-02-11T00:20:33.388 INFO:teuthology.orchestra.run.smithi095.stdout:Building dependency tree... 2021-02-11T00:20:33.388 INFO:teuthology.orchestra.run.smithi095.stdout:Reading state information... 2021-02-11T00:20:33.469 INFO:teuthology.orchestra.run.smithi095.stderr:E: Unable to locate package linux-image-current-generic 2021-02-11T00:20:33.471 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-02-11T00:20:33.471 DEBUG:teuthology.orchestra.run.smithi095:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-02-11T00:20:33.553 INFO:teuthology.orchestra.run.smithi095.stdout:Reading package lists... 2021-02-11T00:20:33.794 INFO:teuthology.orchestra.run.smithi095.stdout:Building dependency tree... 2021-02-11T00:20:33.794 INFO:teuthology.orchestra.run.smithi095.stdout:Reading state information... 2021-02-11T00:20:33.991 INFO:teuthology.orchestra.run.smithi095.stdout:linux-image-generic is already the newest version (4.15.0.135.122). 2021-02-11T00:20:33.992 INFO:teuthology.orchestra.run.smithi095.stdout:0 upgraded, 0 newly installed, 0 to remove and 23 not upgraded. 2021-02-11T00:20:33.993 DEBUG:teuthology.orchestra.run.smithi095:> dpkg -s linux-image-generic 2021-02-11T00:20:34.013 INFO:teuthology.orchestra.run.smithi095.stdout:Package: linux-image-generic 2021-02-11T00:20:34.014 INFO:teuthology.orchestra.run.smithi095.stdout:Status: install ok installed 2021-02-11T00:20:34.014 INFO:teuthology.orchestra.run.smithi095.stdout:Priority: optional 2021-02-11T00:20:34.014 INFO:teuthology.orchestra.run.smithi095.stdout:Section: kernel 2021-02-11T00:20:34.014 INFO:teuthology.orchestra.run.smithi095.stdout:Installed-Size: 17 2021-02-11T00:20:34.014 INFO:teuthology.orchestra.run.smithi095.stdout:Maintainer: Ubuntu Kernel Team 2021-02-11T00:20:34.014 INFO:teuthology.orchestra.run.smithi095.stdout:Architecture: amd64 2021-02-11T00:20:34.014 INFO:teuthology.orchestra.run.smithi095.stdout:Source: linux-meta 2021-02-11T00:20:34.014 INFO:teuthology.orchestra.run.smithi095.stdout:Version: 4.15.0.135.122 2021-02-11T00:20:34.014 INFO:teuthology.orchestra.run.smithi095.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-02-11T00:20:34.015 INFO:teuthology.orchestra.run.smithi095.stdout:Depends: linux-image-4.15.0-135-generic, linux-modules-extra-4.15.0-135-generic, linux-firmware, intel-microcode, amd64-microcode 2021-02-11T00:20:34.015 INFO:teuthology.orchestra.run.smithi095.stdout:Recommends: thermald 2021-02-11T00:20:34.015 INFO:teuthology.orchestra.run.smithi095.stdout:Description: Generic Linux kernel image 2021-02-11T00:20:34.015 INFO:teuthology.orchestra.run.smithi095.stdout: This package will always depend on the latest generic kernel image 2021-02-11T00:20:34.015 INFO:teuthology.orchestra.run.smithi095.stdout: available. 2021-02-11T00:20:34.015 DEBUG:teuthology.orchestra.run.smithi095:> mktemp 2021-02-11T00:20:34.059 INFO:teuthology.orchestra.run.smithi095.stdout:/tmp/tmp.7LsNKlw8S0 2021-02-11T00:20:34.060 DEBUG:teuthology.orchestra.run.smithi095:> sudo cp /boot/grub/grub.cfg /tmp/tmp.7LsNKlw8S0 2021-02-11T00:20:34.115 DEBUG:teuthology.orchestra.run.smithi095:> sudo chmod 0666 /tmp/tmp.7LsNKlw8S0 2021-02-11T00:20:34.239 DEBUG:teuthology.orchestra.remote:smithi095:/tmp/tmp.7LsNKlw8S0 is 8KB 2021-02-11T00:20:34.250 DEBUG:teuthology.orchestra.run.smithi095:> rm -fr /tmp/tmp.7LsNKlw8S0 2021-02-11T00:20:34.255 DEBUG:teuthology.orchestra.run.smithi095:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-02-11T00:20:34.311 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:20:34.312 DEBUG:teuthology.orchestra.run.smithi095:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-02-11T00:20:34.312 DEBUG:teuthology.orchestra.run.smithi095:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-02-11T00:20:34.388 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-135-generic 2021-02-11T00:20:34.388 DEBUG:teuthology.orchestra.run.smithi095:> sudo update-grub 2021-02-11T00:20:35.836 INFO:teuthology.orchestra.run.smithi095.stderr:Sourcing file `/etc/default/grub' 2021-02-11T00:20:35.860 INFO:teuthology.orchestra.run.smithi095.stderr:Generating grub configuration file ... 2021-02-11T00:20:36.112 INFO:teuthology.orchestra.run.smithi095.stderr:Found linux image: /boot/vmlinuz-4.15.0-135-generic 2021-02-11T00:20:36.125 INFO:teuthology.orchestra.run.smithi095.stderr:Found initrd image: /boot/initrd.img-4.15.0-135-generic 2021-02-11T00:20:36.509 INFO:teuthology.orchestra.run.smithi095.stderr:Found linux image: /boot/vmlinuz-4.15.0-134-generic 2021-02-11T00:20:36.522 INFO:teuthology.orchestra.run.smithi095.stderr:Found initrd image: /boot/initrd.img-4.15.0-134-generic 2021-02-11T00:20:37.159 INFO:teuthology.orchestra.run.smithi095.stderr:done 2021-02-11T00:20:37.161 DEBUG:teuthology.orchestra.run.smithi095:> sudo shutdown -r now 2021-02-11T00:21:07.190 INFO:teuthology.misc:Re-opening connections... 2021-02-11T00:21:07.191 INFO:teuthology.misc:trying to connect to ubuntu@smithi057.front.sepia.ceph.com 2021-02-11T00:21:07.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T00:21:07.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T00:21:25.570 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2021-02-11T00:21:25.571 INFO:teuthology.misc:trying to connect to ubuntu@smithi095.front.sepia.ceph.com 2021-02-11T00:21:25.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T00:21:25.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi095.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T00:22:25.574 DEBUG:teuthology.orchestra.remote:timed out 2021-02-11T00:22:25.575 DEBUG:teuthology.misc:waited 78.38389658927917 2021-02-11T00:22:26.575 INFO:teuthology.misc:trying to connect to ubuntu@smithi057.front.sepia.ceph.com 2021-02-11T00:22:26.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T00:22:26.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T00:22:26.651 DEBUG:teuthology.orchestra.run.smithi057:> true 2021-02-11T00:22:27.095 DEBUG:teuthology.misc:waited 79.90421032905579 2021-02-11T00:22:28.096 INFO:teuthology.misc:trying to connect to ubuntu@smithi095.front.sepia.ceph.com 2021-02-11T00:22:28.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T00:22:28.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi095.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T00:22:28.180 DEBUG:teuthology.orchestra.run.smithi095:> true 2021-02-11T00:22:28.650 DEBUG:teuthology.misc:waited 81.45932745933533 2021-02-11T00:22:29.650 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2021-02-11T00:22:29.651 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.15.0-135-generic"... 2021-02-11T00:22:29.651 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2021-02-11T00:22:29.657 INFO:teuthology.orchestra.run.smithi057.stdout:4.15.0-135-generic 2021-02-11T00:22:29.658 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-135-generic vs 4.15.0-135-generic 2021-02-11T00:22:29.658 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-02-11T00:22:29.658 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-02-11T00:22:29.658 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2021-02-11T00:22:29.658 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.15.0-135-generic"... 2021-02-11T00:22:29.658 DEBUG:teuthology.orchestra.run.smithi095:> uname -r 2021-02-11T00:22:29.664 INFO:teuthology.orchestra.run.smithi095.stdout:4.15.0-135-generic 2021-02-11T00:22:29.664 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-135-generic vs 4.15.0-135-generic 2021-02-11T00:22:29.664 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-02-11T00:22:29.664 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-02-11T00:22:30.665 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2021-02-11T00:22:30.666 DEBUG:teuthology.orchestra.run.smithi057:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-11T00:22:30.692 INFO:teuthology.orchestra.run.smithi057.stdout:ttyS1 2021-02-11T00:22:30.712 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2021-02-11T00:22:30.712 DEBUG:teuthology.orchestra.run.smithi095:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-11T00:22:30.758 INFO:teuthology.orchestra.run.smithi095.stdout:ttyS1 2021-02-11T00:22:30.788 INFO:teuthology.run_tasks:Running task internal.base... 2021-02-11T00:22:30.796 INFO:teuthology.task.internal:Creating test directory... 2021-02-11T00:22:30.796 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-11T00:22:30.798 DEBUG:teuthology.orchestra.run.smithi095:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-11T00:22:30.804 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-02-11T00:22:31.217 INFO:teuthology.run_tasks:Running task internal.archive... 2021-02-11T00:22:32.490 INFO:teuthology.task.internal:Creating archive directory... 2021-02-11T00:22:32.491 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-11T00:22:32.493 DEBUG:teuthology.orchestra.run.smithi095:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-11T00:22:32.539 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-02-11T00:22:34.403 INFO:teuthology.task.internal:Enabling coredump saving... 2021-02-11T00:22:34.403 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-11T00:22:34.406 DEBUG:teuthology.orchestra.run.smithi095:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-11T00:22:34.431 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-11T00:22:34.437 INFO:teuthology.orchestra.run.smithi095.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-11T00:22:34.438 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-11T00:22:34.447 INFO:teuthology.orchestra.run.smithi095.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-11T00:22:34.448 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-02-11T00:22:37.762 INFO:teuthology.task.internal:Configuring sudo... 2021-02-11T00:22:37.762 DEBUG:teuthology.orchestra.run.smithi057:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-11T00:22:37.765 DEBUG:teuthology.orchestra.run.smithi095:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-11T00:22:37.780 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-02-11T00:22:37.793 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-02-11T00:22:37.794 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-11T00:22:37.823 DEBUG:teuthology.orchestra.run.smithi095:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-11T00:22:37.829 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-11T00:22:37.875 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-11T00:22:37.928 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:22:37.928 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-11T00:22:37.985 DEBUG:teuthology.orchestra.run.smithi095:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-11T00:22:37.992 DEBUG:teuthology.orchestra.run.smithi095:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-11T00:22:38.040 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:22:38.040 DEBUG:teuthology.orchestra.run.smithi095:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-11T00:22:38.192 DEBUG:teuthology.orchestra.run.smithi057:> sudo service rsyslog restart 2021-02-11T00:22:38.195 DEBUG:teuthology.orchestra.run.smithi095:> sudo service rsyslog restart 2021-02-11T00:22:38.244 INFO:teuthology.run_tasks:Running task internal.timer... 2021-02-11T00:22:38.743 INFO:teuthology.task.internal:Starting timer... 2021-02-11T00:22:38.743 INFO:teuthology.run_tasks:Running task pcp... 2021-02-11T00:22:40.295 INFO:teuthology.run_tasks:Running task selinux... 2021-02-11T00:22:41.509 DEBUG:teuthology.task.selinux:Excluding smithi057: OS 'ubuntu' does not support SELinux 2021-02-11T00:22:41.510 DEBUG:teuthology.task.selinux:Excluding smithi095: OS 'ubuntu' does not support SELinux 2021-02-11T00:22:41.510 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-02-11T00:22:41.510 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2021-02-11T00:22:41.510 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-02-11T00:22:41.510 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-02-11T00:22:42.833 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-02-11T00:22:42.833 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-02-11T00:22:43.093 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-02-11T00:22:43.102 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi057.front.sepia.ceph.com,smithi095.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-02-11T00:22:43.906 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-02-11T00:22:45.842 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-02-11T00:22:48.682 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-02-11T00:22:48.908 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:48 +0000 (0:00:00.158) 0:00:00.158 ***** Thursday 11 February 2021 00:22:48 +0000 (0:00:00.020) 0:00:00.178 ***** 2021-02-11T00:22:50.494 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi057.front.sepia.ceph.com] 2021-02-11T00:22:50.566 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:50 +0000 (0:00:01.814) 0:00:01.993 ***** 2021-02-11T00:22:50.669 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] 2021-02-11T00:22:50.745 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:50 +0000 (0:00:00.178) 0:00:02.171 ***** 2021-02-11T00:22:50.825 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-02-11T00:22:50.826 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-02-11T00:22:50.863 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:50 +0000 (0:00:00.118) 0:00:02.290 ***** 2021-02-11T00:22:50.871 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-02-11T00:22:50.909 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:50 +0000 (0:00:00.045) 0:00:02.335 ***** 2021-02-11T00:22:51.310 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-02-11T00:22:51.312 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-02-11T00:22:51.348 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:51 +0000 (0:00:00.439) 0:00:02.774 ***** 2021-02-11T00:22:51.355 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-02-11T00:22:51.394 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:51 +0000 (0:00:00.046) 0:00:02.821 ***** 2021-02-11T00:22:51.670 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-02-11T00:22:51.672 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.069815", "end": "2021-02-11 00:22:51.890236", "rc": 0, "start": "2021-02-11 00:22:51.820421", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-02-11T00:22:51.706 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:51 +0000 (0:00:00.311) 0:00:03.132 ***** 2021-02-11T00:22:51.711 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.095649", "end": "2021-02-11 00:22:51.637705", "rc": 0, "start": "2021-02-11 00:22:51.542056", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-02-11T00:22:51.748 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:51 +0000 (0:00:00.042) 0:00:03.174 ***** 2021-02-11T00:22:51.950 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-02-11T00:22:51.952 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi057.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T00:22:51.986 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:51 +0000 (0:00:00.238) 0:00:03.413 ***** 2021-02-11T00:22:52.017 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:52 +0000 (0:00:00.030) 0:00:03.443 ***** 2021-02-11T00:22:52.745 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-02-11T00:22:52.746 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613002972.0321143-41680966292546/source", "state": "file", "uid": 0} changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613002971.998517-178979872926874/source", "state": "file", "uid": 0} 2021-02-11T00:22:52.781 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:52 +0000 (0:00:00.764) 0:00:04.208 ***** 2021-02-11T00:22:52.814 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:52 +0000 (0:00:00.032) 0:00:04.240 ***** 2021-02-11T00:22:53.079 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-02-11T00:22:53.080 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-02-11T00:22:53.082 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-02-11T00:22:53.113 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-02-11T00:22:53.115 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-02-11T00:22:53.230 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-02-11T00:22:53.237 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-02-11T00:22:53.387 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:53 +0000 (0:00:00.573) 0:00:04.814 ***** 2021-02-11T00:22:53.422 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:53 +0000 (0:00:00.034) 0:00:04.849 ***** 2021-02-11T00:22:54.075 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-02-11T00:22:54.076 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] 2021-02-11T00:22:54.155 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-02-11T00:22:54.157 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:54 +0000 (0:00:00.733) 0:00:05.582 ***** 2021-02-11T00:22:54.167 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] 2021-02-11T00:22:54.249 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:54 +0000 (0:00:00.093) 0:00:05.676 ***** 2021-02-11T00:22:54.933 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-02-11T00:22:54.934 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.414650", "end": "2021-02-11 00:22:54.897137", "rc": 0, "start": "2021-02-11 00:22:54.482487", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-02-11T00:22:54.968 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:54 +0000 (0:00:00.718) 0:00:06.394 ***** 2021-02-11T00:22:54.975 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.430512", "end": "2021-02-11 00:22:54.912999", "rc": 0, "start": "2021-02-11 00:22:54.482487", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-02-11T00:22:55.015 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:55 +0000 (0:00:00.046) 0:00:06.441 ***** 2021-02-11T00:22:55.114 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-02-11T00:22:55.115 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.004301", "end": "2021-02-11 00:22:55.329717", "failed_when_result": false, "rc": 0, "start": "2021-02-11 00:22:55.325416", "stderr": "", "stderr_lines": [], "stdout": "0caff10a478f2674721da1965013b225f3ea5800", "stdout_lines": ["0caff10a478f2674721da1965013b225f3ea5800"]} 2021-02-11T00:22:55.162 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:55 +0000 (0:00:00.147) 0:00:06.589 ***** 2021-02-11T00:22:55.169 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.015436", "end": "2021-02-11 00:22:55.115744", "failed_when_result": false, "rc": 0, "start": "2021-02-11 00:22:55.100308", "stderr": "", "stderr_lines": [], "stdout": "0caff10a478f2674721da1965013b225f3ea5800", "stdout_lines": ["0caff10a478f2674721da1965013b225f3ea5800"]} 2021-02-11T00:22:55.207 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:55 +0000 (0:00:00.044) 0:00:06.633 ***** 2021-02-11T00:22:55.213 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2021-02-11T00:22:55.213 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:22:55.253 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:55 +0000 (0:00:00.046) 0:00:06.680 ***** 2021-02-11T00:22:55.261 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:22:55.298 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:55 +0000 (0:00:00.045) 0:00:06.725 ***** 2021-02-11T00:22:55.305 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2021-02-11T00:22:55.305 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:22:55.343 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:55 +0000 (0:00:00.044) 0:00:06.770 ***** 2021-02-11T00:22:55.350 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:22:55.388 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:55 +0000 (0:00:00.044) 0:00:06.814 ***** 2021-02-11T00:22:55.394 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2021-02-11T00:22:55.395 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:22:55.435 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:55 +0000 (0:00:00.046) 0:00:06.861 ***** 2021-02-11T00:22:55.440 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:22:55.480 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:55 +0000 (0:00:00.045) 0:00:06.907 ***** 2021-02-11T00:22:55.488 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2021-02-11T00:22:55.489 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:22:55.525 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:55 +0000 (0:00:00.044) 0:00:06.951 ***** 2021-02-11T00:22:55.538 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:22:55.573 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:55 +0000 (0:00:00.048) 0:00:06.999 ***** 2021-02-11T00:22:55.580 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2021-02-11T00:22:55.581 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:22:55.618 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:55 +0000 (0:00:00.044) 0:00:07.045 ***** 2021-02-11T00:22:55.625 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:22:55.662 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:55 +0000 (0:00:00.043) 0:00:07.088 ***** 2021-02-11T00:22:55.751 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-02-11T00:22:55.753 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T00:22:55.789 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:55 +0000 (0:00:00.126) 0:00:07.215 ***** 2021-02-11T00:22:55.805 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T00:22:55.842 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:22:55 +0000 (0:00:00.053) 0:00:07.269 ***** 2021-02-11T00:22:55.973 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-11T00:22:55.975 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-02-11T00:22:56.017 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-02-11T00:22:56.112 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-02-11T00:22:56.144 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-02-11T00:22:56.260 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-02-11T00:22:56.271 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-02-11T00:22:56.408 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-02-11T00:22:56.409 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-02-11T00:22:56.535 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-02-11T00:22:56.547 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-02-11T00:22:56.839 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-02-11T00:22:56.840 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-02-11T00:22:57.030 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-02-11T00:22:57.042 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-02-11T00:22:57.170 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-02-11T00:22:57.183 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-02-11T00:22:57.299 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-11T00:22:57.322 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-11T00:22:57.429 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-11T00:22:57.462 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-11T00:22:57.567 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-11T00:22:57.609 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-11T00:22:57.683 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-11T00:22:57.757 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-11T00:22:57.810 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-11T00:22:57.905 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-11T00:22:57.937 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-02-11T00:22:58.032 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-02-11T00:22:58.064 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-02-11T00:22:58.160 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-02-11T00:22:58.193 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-02-11T00:22:58.299 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-02-11T00:22:58.312 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-02-11T00:22:58.451 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-02-11T00:22:58.452 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-02-11T00:22:58.567 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-02-11T00:22:58.589 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-02-11T00:22:58.694 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-02-11T00:22:58.737 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-02-11T00:22:58.822 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-11T00:22:58.875 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-02-11T00:22:58.948 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-11T00:22:59.012 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-11T00:22:59.075 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-11T00:22:59.159 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-11T00:22:59.202 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-11T00:22:59.308 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-11T00:22:59.330 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-11T00:22:59.445 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-11T00:22:59.456 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-11T00:22:59.581 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-11T00:22:59.593 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-11T00:22:59.707 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-11T00:22:59.728 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-11T00:22:59.833 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-11T00:22:59.875 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-11T00:22:59.959 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-11T00:23:00.001 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-11T00:23:00.087 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-11T00:23:00.167 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-11T00:23:00.225 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-11T00:23:00.247 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-11T00:23:00.352 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-11T00:23:00.394 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-11T00:23:00.479 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-11T00:23:00.532 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-11T00:23:00.605 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-11T00:23:00.679 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-11T00:23:00.732 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-11T00:23:00.817 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-11T00:23:00.859 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-11T00:23:00.963 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-11T00:23:00.984 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-11T00:23:01.099 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-11T00:23:01.111 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-11T00:23:01.239 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-11T00:23:01.251 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-11T00:23:01.372 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-11T00:23:01.417 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-11T00:23:01.501 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-11T00:23:01.605 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-11T00:23:01.628 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-02-11T00:23:01.755 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-11T00:23:01.768 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-11T00:23:01.885 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-11T00:23:01.898 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-11T00:23:02.022 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-11T00:23:02.023 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-11T00:23:02.159 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-11T00:23:02.171 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-02-11T00:23:02.275 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-11T00:23:02.307 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-11T00:23:02.401 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-11T00:23:02.434 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-11T00:23:02.528 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-11T00:23:02.581 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-11T00:23:02.696 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-11T00:23:02.718 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-11T00:23:02.823 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-11T00:23:02.855 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-11T00:23:02.949 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-11T00:23:02.992 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-11T00:23:03.076 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-11T00:23:03.139 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-11T00:23:03.207 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-11T00:23:03.281 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-11T00:23:03.333 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-11T00:23:03.418 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-11T00:23:03.460 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-11T00:23:03.565 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-11T00:23:03.587 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-11T00:23:03.702 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-11T00:23:03.714 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-11T00:23:03.839 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-11T00:23:03.841 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:03.851 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-11T00:23:03.966 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-11T00:23:03.966 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-11T00:23:04.092 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-11T00:23:04.093 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-11T00:23:04.229 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-11T00:23:04.230 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-11T00:23:04.355 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-11T00:23:04.356 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-11T00:23:04.482 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-11T00:23:04.493 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-11T00:23:04.598 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-11T00:23:04.620 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-11T00:23:04.725 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-11T00:23:04.747 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-11T00:23:04.851 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-11T00:23:04.873 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-11T00:23:04.978 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-11T00:23:05.011 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-11T00:23:05.105 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-11T00:23:05.147 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-11T00:23:05.235 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-11T00:23:05.268 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-11T00:23:05.363 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-11T00:23:05.396 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-11T00:23:05.491 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-11T00:23:05.533 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-11T00:23:05.535 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:05.628 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-11T00:23:05.661 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-11T00:23:05.755 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-11T00:23:05.787 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-11T00:23:05.872 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-11T00:23:05.936 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-11T00:23:06.000 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-11T00:23:06.084 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-11T00:23:06.138 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-11T00:23:06.214 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-11T00:23:06.268 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-11T00:23:06.363 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-11T00:23:06.396 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-11T00:23:06.501 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-11T00:23:06.522 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-11T00:23:06.628 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-11T00:23:06.640 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-11T00:23:06.765 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-11T00:23:06.777 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-11T00:23:06.903 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-11T00:23:06.904 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-11T00:23:07.020 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-11T00:23:07.033 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-11T00:23:07.149 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-11T00:23:07.171 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-11T00:23:07.267 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-11T00:23:07.289 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-11T00:23:07.394 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-11T00:23:07.438 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-11T00:23:07.524 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-11T00:23:07.576 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-11T00:23:07.650 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-02-11T00:23:07.703 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-11T00:23:07.776 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-11T00:23:07.851 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-11T00:23:07.894 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-11T00:23:07.989 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-11T00:23:08.021 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-02-11T00:23:08.136 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-11T00:23:08.148 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-02-11T00:23:08.264 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-11T00:23:08.275 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-11T00:23:08.401 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-11T00:23:08.413 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-11T00:23:08.527 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-11T00:23:08.539 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-02-11T00:23:08.654 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-11T00:23:08.676 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-11T00:23:08.781 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-02-11T00:23:08.804 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-11T00:23:08.898 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-11T00:23:08.941 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-02-11T00:23:09.035 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-02-11T00:23:09.078 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-02-11T00:23:09.152 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-11T00:23:09.206 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-11T00:23:09.283 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-02-11T00:23:09.347 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-11T00:23:09.410 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-02-11T00:23:09.485 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-11T00:23:09.538 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-02-11T00:23:09.602 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-11T00:23:09.665 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-11T00:23:09.729 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-02-11T00:23:09.792 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-11T00:23:09.856 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-11T00:23:09.919 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-11T00:23:10.003 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-02-11T00:23:10.057 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-11T00:23:10.141 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-11T00:23:10.183 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-02-11T00:23:10.278 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-02-11T00:23:10.300 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-11T00:23:10.404 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-02-11T00:23:10.426 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-11T00:23:10.552 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-02-11T00:23:10.553 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-02-11T00:23:10.679 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-11T00:23:10.680 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-11T00:23:10.805 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-11T00:23:10.826 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-11T00:23:10.931 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-11T00:23:10.964 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-11T00:23:11.058 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-11T00:23:11.100 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-11T00:23:11.184 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-11T00:23:11.248 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-02-11T00:23:11.313 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-11T00:23:11.397 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-11T00:23:11.439 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-11T00:23:11.536 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-11T00:23:11.568 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-11T00:23:11.683 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-02-11T00:23:11.694 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-02-11T00:23:11.820 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-11T00:23:11.821 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-02-11T00:23:11.947 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-11T00:23:11.958 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-11T00:23:12.062 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-11T00:23:12.108 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-11T00:23:12.192 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-11T00:23:12.255 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-11T00:23:12.318 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-11T00:23:12.381 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-11T00:23:12.444 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-11T00:23:12.518 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-11T00:23:12.571 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-11T00:23:12.666 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-11T00:23:12.728 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-02-11T00:23:12.805 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-11T00:23:12.837 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-11T00:23:12.941 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-02-11T00:23:12.963 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-11T00:23:13.078 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-02-11T00:23:13.089 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-11T00:23:13.214 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-11T00:23:13.226 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-11T00:23:13.344 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-02-11T00:23:13.377 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-11T00:23:13.474 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-11T00:23:13.570 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-11T00:23:13.600 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-02-11T00:23:13.663 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-11T00:23:13.726 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-02-11T00:23:13.800 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-11T00:23:13.936 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-11T00:23:14.071 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-02-11T00:23:14.217 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-11T00:23:14.513 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} changed: [smithi095.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-02-11T00:23:14.520 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:14 +0000 (0:00:18.678) 0:00:25.947 ***** 2021-02-11T00:23:14.530 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-11T00:23:14.547 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2021-02-11T00:23:14.587 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:14 +0000 (0:00:00.066) 0:00:26.013 ***** 2021-02-11T00:23:14.642 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2021-02-11T00:23:14.646 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-02-11T00:23:14.647 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-02-11T00:23:14.648 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancic 2021-02-11T00:23:14.650 INFO:teuthology.task.ansible.out:h", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1Wi 2021-02-11T00:23:14.651 INFO:teuthology.task.ansible.out:RenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vum 2021-02-11T00:23:14.652 INFO:teuthology.task.ansible.out:rao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf 2021-02-11T00:23:14.652 INFO:teuthology.task.ansible.out:542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69 2021-02-11T00:23:14.654 INFO:teuthology.task.ansible.out:mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d8 2021-02-11T00:23:14.655 INFO:teuthology.task.ansible.out:41a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveirada 2021-02-11T00:23:14.656 INFO:teuthology.task.ansible.out:n@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOe 2021-02-11T00:23:14.658 INFO:teuthology.task.ansible.out:nVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9 2021-02-11T00:23:14.659 INFO:teuthology.task.ansible.out:gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": 2021-02-11T00:23:14.660 INFO:teuthology.task.ansible.out:"dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NI 2021-02-11T00:23:14.662 INFO:teuthology.task.ansible.out:Nj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395 2021-02-11T00:23:14.662 INFO:teuthology.task.ansible.out:862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-02-11T00:23:14.665 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-11T00:23:14.665 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-11T00:23:14.720 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:14 +0000 (0:00:00.132) 0:00:26.146 ***** 2021-02-11T00:23:14.780 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-02-11T00:23:14.918 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-11T00:23:15.067 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-02-11T00:23:16.349 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1237} TASK [users : Clone the keys repo] ********************************************* ok: [smithi095.front.sepia.ceph.com -> localhost] => {"after": "df8946f08e870e2d2565b79099289ca48bd825f3", "attempts": 1, "before": "df8946f08e870e2d2565b79099289ca48bd825f3", "changed": false, "remote_url_changed": false} Thursday 11 February 2021 00:23:15 +0000 (0:00:00.924) 0:00:27.071 ***** TASK [users : Create all admin users with sudo access.] ************************ changed: [smithi057.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"n 2021-02-11T00:23:16.350 INFO:teuthology.task.ansible.out:ame": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} Thursday 11 February 2021 00:23:15 +0000 (0:00:00.224) 0:00:27.295 ***** TASK [users : Create all users without sudo access.] *************************** Thursday 11 February 2021 00:23:15 +0000 (0:00:00.054) 0:00:27.349 ***** TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf 2021-02-11T00:23:16.352 INFO:teuthology.task.ansible.out:7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} TASK [users : Merge managed_users and managed_admin users] ********************* ok: [smithi057.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b4204 2021-02-11T00:23:16.353 INFO:teuthology.task.ansible.out:2aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e 2021-02-11T00:23:16.354 INFO:teuthology.task.ansible.out:1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa 2021-02-11T00:23:16.355 INFO:teuthology.task.ansible.out:6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPs 2021-02-11T00:23:16.357 INFO:teuthology.task.ansible.out:pffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a 2021-02-11T00:23:16.357 INFO:teuthology.task.ansible.out:0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec94 2021-02-11T00:23:16.358 INFO:teuthology.task.ansible.out:67ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDj 2021-02-11T00:23:16.360 INFO:teuthology.task.ansible.out:w 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrio 2021-02-11T00:23:16.361 INFO:teuthology.task.ansible.out:ux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaari 2021-02-11T00:23:16.362 INFO:teuthology.task.ansible.out:t@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "o 2021-02-11T00:23:16.362 INFO:teuthology.task.ansible.out:vpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956 2021-02-11T00:23:16.364 INFO:teuthology.task.ansible.out:c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn" 2021-02-11T00:23:16.365 INFO:teuthology.task.ansible.out:: "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {" 2021-02-11T00:23:16.366 INFO:teuthology.task.ansible.out:name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc3 2021-02-11T00:23:16.367 INFO:teuthology.task.ansible.out:4497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395 2021-02-11T00:23:16.368 INFO:teuthology.task.ansible.out:862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc67375 2021-02-11T00:23:16.369 INFO:teuthology.task.ansible.out:7267"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-02-11T00:23:16.371 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:16.372 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d04521 2021-02-11T00:23:16.374 INFO:teuthology.task.ansible.out:66af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 2021-02-11T00:23:16.375 INFO:teuthology.task.ansible.out:09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-11T00:23:16.409 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:16 +0000 (0:00:00.486) 0:00:27.836 ***** 2021-02-11T00:23:16.459 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-02-11T00:23:16.460 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-11T00:23:16.585 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-11T00:23:16.713 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-11T00:23:16.886 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-11T00:23:16.897 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:17.023 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-02-11T00:23:17.025 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-02-11T00:23:17.034 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-02-11T00:23:17.036 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com -> localhost] => {"after": "df8946f08e870e2d2565b79099289ca48bd825f3", "attempts": 1, "before": "df8946f08e870e2d2565b79099289ca48bd825f3", "changed": false, "remote_url_changed": false} 2021-02-11T00:23:17.069 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:17 +0000 (0:00:00.659) 0:00:28.496 ***** 2021-02-11T00:23:17.151 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:17.152 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-02-11T00:23:17.154 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-02-11T00:23:17.162 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:17.173 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:17.269 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-02-11T00:23:17.291 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-11T00:23:17.417 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-11T00:23:17.429 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-02-11T00:23:17.430 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-11T00:23:17.544 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-11T00:23:17.555 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-11T00:23:17.670 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-11T00:23:17.692 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-02-11T00:23:17.694 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-11T00:23:17.796 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-11T00:23:17.818 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-11T00:23:17.912 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-11T00:23:17.945 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-02-11T00:23:17.946 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-11T00:23:18.029 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-11T00:23:18.092 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-11T00:23:18.145 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-02-11T00:23:18.147 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-02-11T00:23:18.229 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-11T00:23:18.261 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-02-11T00:23:18.262 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-02-11T00:23:18.377 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-11T00:23:18.388 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:18.388 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-11T00:23:18.514 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-02-11T00:23:18.515 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-11T00:23:18.630 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-02-11T00:23:18.632 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-02-11T00:23:18.662 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-02-11T00:23:18.664 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-02-11T00:23:18.673 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:18.684 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:18.758 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-02-11T00:23:18.810 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-11T00:23:18.873 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-02-11T00:23:18.946 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-02-11T00:23:18.947 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-11T00:23:18.988 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-11T00:23:19.072 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-11T00:23:19.115 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-11T00:23:19.208 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-11T00:23:19.230 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-11T00:23:19.346 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-11T00:23:19.357 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-02-11T00:23:19.359 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-02-11T00:23:19.472 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-11T00:23:19.494 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-11T00:23:19.608 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-11T00:23:19.609 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-11T00:23:19.723 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-11T00:23:19.745 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-02-11T00:23:19.746 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-02-11T00:23:19.839 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-11T00:23:19.892 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-02-11T00:23:19.893 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-02-11T00:23:19.965 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-11T00:23:20.008 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-11T00:23:20.091 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-11T00:23:20.154 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-11T00:23:20.217 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-11T00:23:20.290 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-02-11T00:23:20.292 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-02-11T00:23:20.333 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-11T00:23:20.428 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-02-11T00:23:20.449 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-11T00:23:20.564 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-02-11T00:23:20.565 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-11T00:23:20.575 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-02-11T00:23:20.689 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-02-11T00:23:20.691 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-02-11T00:23:20.711 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-11T00:23:20.805 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-11T00:23:20.847 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-11T00:23:20.920 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-11T00:23:20.973 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-11T00:23:21.036 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-11T00:23:21.088 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-02-11T00:23:21.090 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-02-11T00:23:21.151 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-02-11T00:23:21.153 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-11T00:23:21.235 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-11T00:23:21.277 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-02-11T00:23:21.278 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-02-11T00:23:21.280 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-02-11T00:23:21.373 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-11T00:23:21.394 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-11T00:23:21.509 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-11T00:23:21.520 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-11T00:23:21.635 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-11T00:23:21.656 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-11T00:23:21.750 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-11T00:23:21.792 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-11T00:23:21.865 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-02-11T00:23:21.928 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-11T00:23:21.981 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-02-11T00:23:21.982 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-02-11T00:23:22.074 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-11T00:23:22.096 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-11T00:23:22.200 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-11T00:23:22.222 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-11T00:23:22.336 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-11T00:23:22.337 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-02-11T00:23:22.338 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-02-11T00:23:22.452 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-11T00:23:22.473 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-02-11T00:23:22.474 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-11T00:23:22.577 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-02-11T00:23:22.619 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-02-11T00:23:22.620 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-02-11T00:23:22.692 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-02-11T00:23:22.694 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-02-11T00:23:22.776 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-11T00:23:22.808 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-02-11T00:23:22.901 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-11T00:23:22.933 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-11T00:23:23.017 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-11T00:23:23.039 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-11T00:23:23.153 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-11T00:23:23.164 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-02-11T00:23:23.165 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-11T00:23:23.279 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-02-11T00:23:23.311 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-02-11T00:23:23.312 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-02-11T00:23:23.314 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-02-11T00:23:23.389 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-02-11T00:23:23.443 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-11T00:23:23.516 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-02-11T00:23:23.559 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-11T00:23:23.632 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-11T00:23:23.705 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-11T00:23:23.758 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-02-11T00:23:23.842 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-11T00:23:23.874 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-02-11T00:23:23.978 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-02-11T00:23:23.989 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-02-11T00:23:24.104 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-02-11T00:23:24.125 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-02-11T00:23:24.127 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-02-11T00:23:24.229 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-02-11T00:23:24.272 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-11T00:23:24.345 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-02-11T00:23:24.450 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-11T00:23:24.461 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-02-11T00:23:24.575 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-02-11T00:23:24.586 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-02-11T00:23:24.588 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-02-11T00:23:24.691 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-02-11T00:23:24.724 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-11T00:23:24.808 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-02-11T00:23:24.851 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-02-11T00:23:24.935 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-02-11T00:23:24.988 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-02-11T00:23:24.989 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-02-11T00:23:25.061 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-02-11T00:23:25.135 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-02-11T00:23:25.178 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-02-11T00:23:25.179 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-02-11T00:23:25.273 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-11T00:23:25.316 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-02-11T00:23:25.417 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-11T00:23:25.430 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-02-11T00:23:25.546 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-11T00:23:25.547 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-11T00:23:25.663 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-02-11T00:23:25.674 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-02-11T00:23:25.789 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-02-11T00:23:25.791 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-02-11T00:23:25.800 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-02-11T00:23:25.905 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-02-11T00:23:25.907 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-02-11T00:23:25.958 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-02-11T00:23:26.021 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-02-11T00:23:26.095 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-11T00:23:26.137 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-02-11T00:23:26.232 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-02-11T00:23:26.253 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-02-11T00:23:26.378 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-02-11T00:23:26.379 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-02-11T00:23:26.494 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-02-11T00:23:26.526 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-02-11T00:23:26.610 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-02-11T00:23:26.663 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-02-11T00:23:26.726 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-02-11T00:23:26.810 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-02-11T00:23:26.842 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-02-11T00:23:26.844 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-02-11T00:23:26.946 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-02-11T00:23:26.968 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-02-11T00:23:27.084 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-02-11T00:23:27.085 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-02-11T00:23:27.200 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-02-11T00:23:27.232 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-02-11T00:23:27.315 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-02-11T00:23:27.367 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-02-11T00:23:27.431 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-02-11T00:23:27.515 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-02-11T00:23:27.547 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-02-11T00:23:27.651 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-02-11T00:23:27.672 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-02-11T00:23:27.673 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-11T00:23:27.777 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-02-11T00:23:27.791 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-02-11T00:23:27.940 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-02-11T00:23:27.941 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-02-11T00:23:27.961 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-02-11T00:23:27.963 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:28.079 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-02-11T00:23:28.091 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-02-11T00:23:28.208 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-02-11T00:23:28.241 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-02-11T00:23:28.357 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-02-11T00:23:28.417 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-11T00:23:28.472 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-02-11T00:23:28.578 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-02-11T00:23:28.599 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-02-11T00:23:28.714 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-02-11T00:23:28.725 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-02-11T00:23:28.727 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-02-11T00:23:28.830 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-02-11T00:23:28.862 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-02-11T00:23:28.863 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-02-11T00:23:28.946 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-02-11T00:23:29.009 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-02-11T00:23:29.072 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-02-11T00:23:29.145 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-02-11T00:23:29.188 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-02-11T00:23:29.292 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-02-11T00:23:29.314 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-02-11T00:23:29.431 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-02-11T00:23:29.431 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-02-11T00:23:29.546 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-02-11T00:23:29.567 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-02-11T00:23:29.671 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-02-11T00:23:29.713 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-02-11T00:23:29.797 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-02-11T00:23:29.840 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-02-11T00:23:29.914 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-02-11T00:23:29.956 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-02-11T00:23:29.957 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-02-11T00:23:30.040 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-02-11T00:23:30.041 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-02-11T00:23:30.103 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-02-11T00:23:30.155 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-02-11T00:23:30.250 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-02-11T00:23:30.282 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-02-11T00:23:30.387 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-02-11T00:23:30.408 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-02-11T00:23:30.523 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-02-11T00:23:30.535 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-02-11T00:23:30.650 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-02-11T00:23:30.651 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-02-11T00:23:30.776 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-02-11T00:23:30.808 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-02-11T00:23:30.902 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-02-11T00:23:30.923 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-02-11T00:23:30.924 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-11T00:23:31.027 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-02-11T00:23:31.038 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-02-11T00:23:31.133 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-02-11T00:23:31.185 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-02-11T00:23:31.187 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:31.259 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-02-11T00:23:31.312 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-02-11T00:23:31.385 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-02-11T00:23:31.443 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-02-11T00:23:31.516 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-02-11T00:23:31.590 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-02-11T00:23:31.633 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-02-11T00:23:31.726 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-02-11T00:23:31.758 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-02-11T00:23:31.863 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-02-11T00:23:31.874 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-02-11T00:23:31.979 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-02-11T00:23:32.001 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-02-11T00:23:32.095 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-02-11T00:23:32.117 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-02-11T00:23:32.231 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-02-11T00:23:32.253 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-02-11T00:23:32.331 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:32 +0000 (0:00:15.262) 0:00:43.758 ***** 2021-02-11T00:23:32.370 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-02-11T00:23:32.371 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T00:23:32.371 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.382 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.393 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.394 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.405 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.417 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.418 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.430 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.441 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.494 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:32.495 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-02-11T00:23:32.569 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T00:23:32.570 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-02-11T00:23:32.581 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.593 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.625 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:32.626 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-02-11T00:23:32.721 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T00:23:32.722 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-11T00:23:32.743 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:32.744 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-02-11T00:23:32.837 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T00:23:32.839 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-02-11T00:23:32.849 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.849 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.860 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.871 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.871 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:32.872 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.883 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.894 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.905 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.905 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.916 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.917 INFO:teuthology.task.ansible.out: skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.927 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.929 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:32.939 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.950 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.951 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.962 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.973 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.974 INFO:teuthology.task.ansible.out: skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.984 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.985 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:32.995 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:32.996 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.007 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:33.008 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-02-11T00:23:33.009 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.019 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.021 INFO:teuthology.task.ansible.out: skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.031 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.032 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:33.042 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.043 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.054 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.065 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.066 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.076 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.087 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.087 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.098 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.109 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.120 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.130 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.131 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.142 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.142 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:33.143 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.154 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.165 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.176 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.176 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.187 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.198 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.198 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.209 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.220 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.220 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.232 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.242 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.243 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.254 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.265 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:33.266 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.276 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.277 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.288 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.298 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.299 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.310 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.320 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.321 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.332 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.343 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.344 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.355 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.365 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.366 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.377 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.388 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.399 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.399 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.410 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:33.412 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-02-11T00:23:33.413 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.433 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.433 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.435 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:33.455 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.456 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:33.466 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.477 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.488 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.490 INFO:teuthology.task.ansible.out: skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.500 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.511 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.511 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.522 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.534 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:33.535 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.535 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.546 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.557 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.558 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.569 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.580 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.581 INFO:teuthology.task.ansible.out: skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.592 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.593 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:33.603 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.614 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.616 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:33.617 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.627 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.627 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.639 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.650 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.651 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:33.651 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-02-11T00:23:33.652 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.662 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.673 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.674 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.685 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.696 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.707 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.707 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.718 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.729 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.729 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.740 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.751 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.752 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.763 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.773 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.774 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:33.774 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-02-11T00:23:33.776 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.785 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.797 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.807 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.808 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.819 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.830 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.830 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.882 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:33 +0000 (0:00:01.550) 0:00:45.309 ***** 2021-02-11T00:23:33.909 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:33.911 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:33.948 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:33 +0000 (0:00:00.065) 0:00:45.374 ***** 2021-02-11T00:23:34.041 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:34.041 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-02-11T00:23:34.115 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:34.116 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-11T00:23:34.178 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:34.179 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-02-11T00:23:34.231 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:34.232 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-11T00:23:34.305 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:34.306 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-02-11T00:23:34.358 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:34.359 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-11T00:23:34.442 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:34.444 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-02-11T00:23:34.475 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:34.476 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-11T00:23:34.569 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:34.570 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-02-11T00:23:34.601 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:34.602 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-11T00:23:34.695 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:34.697 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-02-11T00:23:34.717 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:34.718 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-11T00:23:34.841 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:34.843 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} TASK [users : Remove revoked users] ******************************************** ok: [smithi095.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-11T00:23:34.957 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-02-11T00:23:34.968 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:34.970 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-02-11T00:23:35.086 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:35.087 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-02-11T00:23:35.097 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:35.098 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-02-11T00:23:35.201 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:35.202 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-02-11T00:23:35.222 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:35.224 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-02-11T00:23:35.327 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi095.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-11T00:23:35.348 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:35.350 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-02-11T00:23:35.442 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:35.444 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-11T00:23:35.475 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:35.476 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-02-11T00:23:35.579 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:35.581 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-11T00:23:35.622 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:35.623 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-02-11T00:23:35.696 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:35.697 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-02-11T00:23:35.738 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-11T00:23:35.740 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-02-11T00:23:35.827 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:35 +0000 (0:00:01.879) 0:00:47.254 ***** 2021-02-11T00:23:35.835 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:35.837 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-11T00:23:35.871 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T00:23:35.872 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:35.882 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:35.884 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:35.885 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:35.895 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:35.897 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:35.908 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:35.909 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:35.919 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:35.921 INFO:teuthology.task.ansible.out: skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:35.931 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:35.933 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi095.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-02-11T00:23:35.944 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:35.946 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.052 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:36.053 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-02-11T00:23:36.074 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T00:23:36.075 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-02-11T00:23:36.085 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.086 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.181 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:36.182 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-11T00:23:36.224 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T00:23:36.225 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-11T00:23:36.298 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:36.299 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-11T00:23:36.342 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi057.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-02-11T00:23:36.353 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.354 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:36.364 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.375 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.386 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.397 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.408 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.419 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.421 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi095.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.432 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.443 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.454 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.456 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.466 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.477 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.478 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.489 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.500 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.500 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.512 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.513 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.524 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.536 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.536 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:36.538 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-11T00:23:36.548 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T00:23:36.549 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.559 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.560 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.570 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.581 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.582 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.593 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.604 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.605 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.615 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.626 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.637 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.638 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.648 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.649 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.660 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:36.662 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.672 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.673 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.684 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.695 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.696 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.707 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.718 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.719 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.730 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.741 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.742 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.753 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.765 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.776 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.778 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.780 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:36.781 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-11T00:23:36.793 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T00:23:36.794 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.805 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.805 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.816 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.827 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.839 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.849 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.851 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.861 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.871 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.883 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.883 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.894 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.895 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:36.895 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-11T00:23:36.906 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T00:23:36.908 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.918 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.918 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.929 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.940 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.951 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.952 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.962 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.973 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.974 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.984 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:36.995 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.006 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.007 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.018 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.018 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:37.019 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-11T00:23:37.029 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-11T00:23:37.031 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.041 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.051 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.062 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.063 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.073 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.084 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.095 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.096 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.106 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.118 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.128 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.130 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.139 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.140 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:37.141 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.152 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.163 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.174 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.174 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.185 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.196 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.196 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.207 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.218 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.230 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.230 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.241 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.252 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.253 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.263 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:37.264 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.275 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.275 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.286 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.297 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.297 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.308 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.319 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.330 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.330 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.342 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.395 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:37 +0000 (0:00:01.567) 0:00:48.822 ***** 2021-02-11T00:23:37.404 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:37.405 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-11T00:23:37.417 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2021-02-11T00:23:37.418 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:37.453 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:37 +0000 (0:00:00.058) 0:00:48.880 ***** 2021-02-11T00:23:37.503 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:37.505 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-02-11T00:23:37.629 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-11T00:23:37.630 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-02-11T00:23:37.745 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-11T00:23:37.766 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-11T00:23:37.874 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-11T00:23:37.998 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-11T00:23:38.012 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-11T00:23:38.033 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-11T00:23:38.141 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-11T00:23:38.153 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-11T00:23:38.267 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-02-11T00:23:38.289 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-11T00:23:38.393 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-11T00:23:38.404 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-11T00:23:38.509 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-11T00:23:38.541 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-02-11T00:23:38.635 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-02-11T00:23:38.657 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-02-11T00:23:38.771 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-02-11T00:23:38.907 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-11T00:23:39.021 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-11T00:23:39.136 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-02-11T00:23:39.157 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-11T00:23:39.261 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-11T00:23:39.263 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:39.283 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-02-11T00:23:39.284 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:39.388 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-11T00:23:39.410 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-11T00:23:39.506 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-11T00:23:39.517 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-02-11T00:23:39.632 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-11T00:23:39.653 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-02-11T00:23:39.747 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-11T00:23:39.789 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-11T00:23:39.863 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-11T00:23:39.905 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-11T00:23:39.989 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-11T00:23:40.034 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-02-11T00:23:40.107 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-11T00:23:40.160 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-11T00:23:40.223 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-11T00:23:40.286 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-02-11T00:23:40.350 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-11T00:23:40.406 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:40 +0000 (0:00:02.952) 0:00:51.832 ***** 2021-02-11T00:23:40.412 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-11T00:23:40.528 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-11T00:23:40.654 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-11T00:23:40.738 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-02-11T00:23:40.739 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ee06ed012f51c2e3764171d4b9180349", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613003020.4302895-97001495212343/source", "state": "file", "uid": 0} 2021-02-11T00:23:40.774 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:40 +0000 (0:00:00.368) 0:00:52.201 ***** 2021-02-11T00:23:40.782 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:40.783 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-11T00:23:40.804 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-02-11T00:23:40.805 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => { "msg": "Host smithi095.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-02-11T00:23:40.840 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:40 +0000 (0:00:00.065) 0:00:52.267 ***** 2021-02-11T00:23:40.878 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-02-11T00:23:40.879 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-02-11T00:23:40.914 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:40 +0000 (0:00:00.073) 0:00:52.340 ***** 2021-02-11T00:23:40.920 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:40.921 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-02-11T00:23:40.933 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-02-11T00:23:40.934 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:40.968 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:40 +0000 (0:00:00.053) 0:00:52.394 ***** 2021-02-11T00:23:40.995 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:41.030 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:41 +0000 (0:00:00.062) 0:00:52.456 ***** 2021-02-11T00:23:41.036 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:41.036 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-11T00:23:41.058 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-02-11T00:23:41.059 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:41.095 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:41 +0000 (0:00:00.065) 0:00:52.522 ***** 2021-02-11T00:23:41.122 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-02-11T00:23:41.123 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:41.176 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:41 +0000 (0:00:00.080) 0:00:52.602 ***** 2021-02-11T00:23:41.183 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:41.183 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-02-11T00:23:41.205 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:41.241 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:41 +0000 (0:00:00.065) 0:00:52.668 ***** 2021-02-11T00:23:41.258 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:41.259 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-02-11T00:23:41.270 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:41.305 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:41 +0000 (0:00:00.063) 0:00:52.732 ***** 2021-02-11T00:23:41.331 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-02-11T00:23:41.333 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:41.366 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:41 +0000 (0:00:00.061) 0:00:52.793 ***** 2021-02-11T00:23:41.383 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:41.384 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-11T00:23:41.394 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-02-11T00:23:41.396 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:41.431 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:41 +0000 (0:00:00.065) 0:00:52.858 ***** 2021-02-11T00:23:41.470 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-02-11T00:23:41.472 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi095.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T00:23:41.517 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:41 +0000 (0:00:00.085) 0:00:52.944 ***** 2021-02-11T00:23:41.524 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:41.525 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-11T00:23:41.547 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-02-11T00:23:41.548 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:41.585 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:41 +0000 (0:00:00.067) 0:00:53.011 ***** 2021-02-11T00:23:41.613 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-02-11T00:23:41.615 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:41.685 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:41 +0000 (0:00:00.099) 0:00:53.111 ***** 2021-02-11T00:23:41.696 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:41.698 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-11T00:23:41.723 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi095.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T00:23:41.773 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:41 +0000 (0:00:00.087) 0:00:53.199 ***** 2021-02-11T00:23:41.792 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:41.795 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-11T00:23:41.804 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-02-11T00:23:41.805 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:41.839 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:41 +0000 (0:00:00.066) 0:00:53.266 ***** 2021-02-11T00:23:41.868 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi095.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T00:23:41.924 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:41 +0000 (0:00:00.085) 0:00:53.351 ***** 2021-02-11T00:23:41.932 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:41.932 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-02-11T00:23:41.965 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-02-11T00:23:41.967 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:42.005 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:41 +0000 (0:00:00.080) 0:00:53.431 ***** 2021-02-11T00:23:42.031 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-02-11T00:23:42.033 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:42.072 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:42 +0000 (0:00:00.067) 0:00:53.498 ***** 2021-02-11T00:23:42.078 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:42.079 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-11T00:23:42.101 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:42.146 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:42 +0000 (0:00:00.074) 0:00:53.573 ***** 2021-02-11T00:23:42.175 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-02-11T00:23:42.177 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:42.215 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:42 +0000 (0:00:00.068) 0:00:53.641 ***** 2021-02-11T00:23:42.225 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:42.226 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-11T00:23:42.238 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-02-11T00:23:42.240 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T00:23:42.287 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:42 +0000 (0:00:00.071) 0:00:53.713 ***** 2021-02-11T00:23:42.288 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:42.317 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:42.318 INFO:teuthology.task.ansible.out:TASK [common : Clean yum cache] ************************************************ skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:42.364 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:42 +0000 (0:00:00.076) 0:00:53.790 ***** 2021-02-11T00:23:42.372 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:42.373 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-02-11T00:23:42.396 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-11T00:23:42.397 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:42.431 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:42 +0000 (0:00:00.067) 0:00:53.857 ***** 2021-02-11T00:23:42.465 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-02-11T00:23:42.466 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:42.501 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:42 +0000 (0:00:00.069) 0:00:53.927 ***** 2021-02-11T00:23:42.529 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-11T00:23:42.531 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:42.564 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:42 +0000 (0:00:00.063) 0:00:53.991 ***** 2021-02-11T00:23:42.592 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-11T00:23:42.594 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:42.627 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:42 +0000 (0:00:00.062) 0:00:54.054 ***** 2021-02-11T00:23:42.654 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:42.688 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:42 +0000 (0:00:00.061) 0:00:54.115 ***** 2021-02-11T00:23:42.715 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-02-11T00:23:42.716 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} TASK [users : Remove revoked users] ******************************************** changed: [smithi057.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-02-11T00:23:42.750 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:42 +0000 (0:00:00.062) 0:00:54.177 ***** 2021-02-11T00:23:42.776 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-02-11T00:23:42.778 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:42.811 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:42 +0000 (0:00:00.060) 0:00:54.238 ***** 2021-02-11T00:23:42.839 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-11T00:23:42.840 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:42.873 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:42 +0000 (0:00:00.061) 0:00:54.300 ***** 2021-02-11T00:23:42.879 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:42.879 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-11T00:23:42.901 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-02-11T00:23:42.903 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:42.936 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:42 +0000 (0:00:00.062) 0:00:54.363 ***** 2021-02-11T00:23:42.977 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-02-11T00:23:42.978 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:42.988 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:42.989 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:42.991 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-11T00:23:43.024 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:43 +0000 (0:00:00.088) 0:00:54.451 ***** 2021-02-11T00:23:43.061 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-11T00:23:43.062 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi095.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-02-11T00:23:43.097 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:43 +0000 (0:00:00.072) 0:00:54.523 ***** 2021-02-11T00:23:43.126 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:43.127 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-11T00:23:43.162 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:43 +0000 (0:00:00.065) 0:00:54.589 ***** 2021-02-11T00:23:43.264 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-11T00:23:43.306 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-11T00:23:43.308 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.009554", "end": "2021-02-11 00:23:43.279614", "rc": 0, "start": "2021-02-11 00:23:43.270060", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-02-11T00:23:43.348 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:43 +0000 (0:00:00.184) 0:00:54.774 ***** 2021-02-11T00:23:43.374 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-02-11T00:23:43.376 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:43.416 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:43 +0000 (0:00:00.068) 0:00:54.843 ***** 2021-02-11T00:23:43.424 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:43.424 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-11T00:23:43.447 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-02-11T00:23:43.447 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:43.483 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:43 +0000 (0:00:00.066) 0:00:54.909 ***** 2021-02-11T00:23:43.520 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-02-11T00:23:43.522 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-02-11T00:23:43.533 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-02-11T00:23:43.543 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:43.544 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-11T00:23:43.545 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** skipping: [smithi095.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi095.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-11T00:23:43.581 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:43 +0000 (0:00:00.098) 0:00:55.008 ***** 2021-02-11T00:23:43.608 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-11T00:23:43.610 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:43.644 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:43 +0000 (0:00:00.062) 0:00:55.071 ***** 2021-02-11T00:23:43.670 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:43.672 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} TASK [common : Set the timezone] *********************************************** skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:43.708 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:43 +0000 (0:00:00.063) 0:00:55.135 ***** 2021-02-11T00:23:43.737 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-02-11T00:23:43.739 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:43.775 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:43 +0000 (0:00:00.066) 0:00:55.201 ***** 2021-02-11T00:23:43.792 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:43.792 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-11T00:23:43.804 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-02-11T00:23:43.805 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:43.841 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:43 +0000 (0:00:00.066) 0:00:55.268 ***** 2021-02-11T00:23:43.869 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-02-11T00:23:43.871 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:43.909 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:43 +0000 (0:00:00.068) 0:00:55.336 ***** 2021-02-11T00:23:43.926 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-11T00:23:43.927 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-11T00:23:44.052 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-11T00:23:44.121 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:44 +0000 (0:00:00.211) 0:00:55.548 ***** 2021-02-11T00:23:44.461 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-02-11T00:23:44.462 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ee06ed012f51c2e3764171d4b9180349", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613003024.1450274-39968361590131/source", "state": "file", "uid": 0} 2021-02-11T00:23:44.496 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:44 +0000 (0:00:00.374) 0:00:55.923 ***** 2021-02-11T00:23:44.527 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-02-11T00:23:44.528 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.492142", "end": "2021-02-11 00:23:44.500385", "rc": 0, "start": "2021-02-11 00:23:44.008243", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [common : Log the OS name, version and release] *************************** ok: [smithi057.front.sepia.ceph.com] => { "msg": "Host smithi057.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-02-11T00:23:44.562 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:44 +0000 (0:00:00.066) 0:00:55.989 ***** 2021-02-11T00:23:44.597 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:44 +0000 (0:00:00.034) 0:00:56.023 ***** 2021-02-11T00:23:44.603 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-02-11T00:23:44.603 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-02-11T00:23:44.639 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:44 +0000 (0:00:00.042) 0:00:56.065 ***** 2021-02-11T00:23:44.646 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-02-11T00:23:44.647 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:44.682 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:44 +0000 (0:00:00.042) 0:00:56.108 ***** 2021-02-11T00:23:44.687 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-02-11T00:23:44.688 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:44.724 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:44 +0000 (0:00:00.042) 0:00:56.151 ***** 2021-02-11T00:23:44.751 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-02-11T00:23:44.752 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:44.787 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:44 +0000 (0:00:00.062) 0:00:56.213 ***** 2021-02-11T00:23:44.814 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-02-11T00:23:44.815 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:44.849 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:44 +0000 (0:00:00.062) 0:00:56.276 ***** 2021-02-11T00:23:44.866 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-02-11T00:23:44.868 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.052495", "end": "2021-02-11 00:23:44.833831", "rc": 0, "start": "2021-02-11 00:23:44.781336", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-02-11T00:23:44.868 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:44.903 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:44 +0000 (0:00:00.053) 0:00:56.330 ***** 2021-02-11T00:23:44.936 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:44 +0000 (0:00:00.032) 0:00:56.362 ***** 2021-02-11T00:23:44.943 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-02-11T00:23:44.944 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:44.979 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:44 +0000 (0:00:00.043) 0:00:56.406 ***** 2021-02-11T00:23:45.007 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:45.042 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:45 +0000 (0:00:00.062) 0:00:56.469 ***** 2021-02-11T00:23:45.070 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-02-11T00:23:45.072 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:45.106 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:45 +0000 (0:00:00.063) 0:00:56.532 ***** 2021-02-11T00:23:45.133 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-02-11T00:23:45.135 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:45.170 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:45 +0000 (0:00:00.064) 0:00:56.597 ***** 2021-02-11T00:23:45.212 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-02-11T00:23:45.214 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi057.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T00:23:45.248 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:45 +0000 (0:00:00.077) 0:00:56.674 ***** 2021-02-11T00:23:45.276 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-02-11T00:23:45.277 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:45.314 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:45 +0000 (0:00:00.066) 0:00:56.741 ***** 2021-02-11T00:23:45.342 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:45.376 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:45 +0000 (0:00:00.062) 0:00:56.803 ***** 2021-02-11T00:23:45.403 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-02-11T00:23:45.404 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T00:23:45.438 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:45 +0000 (0:00:00.061) 0:00:56.864 ***** 2021-02-11T00:23:45.466 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-02-11T00:23:45.468 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:45.502 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:45 +0000 (0:00:00.064) 0:00:56.929 ***** 2021-02-11T00:23:45.530 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-02-11T00:23:45.532 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T00:23:45.568 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:45 +0000 (0:00:00.065) 0:00:56.994 ***** 2021-02-11T00:23:45.596 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-02-11T00:23:45.597 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:45.631 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:45 +0000 (0:00:00.063) 0:00:57.058 ***** 2021-02-11T00:23:45.658 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-02-11T00:23:45.660 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:45.694 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:45 +0000 (0:00:00.062) 0:00:57.121 ***** 2021-02-11T00:23:45.722 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-02-11T00:23:45.723 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:45.757 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:45 +0000 (0:00:00.063) 0:00:57.184 ***** 2021-02-11T00:23:45.785 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-02-11T00:23:45.787 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:45.821 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:45 +0000 (0:00:00.063) 0:00:57.247 ***** 2021-02-11T00:23:45.849 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-02-11T00:23:45.851 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T00:23:45.885 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:45 +0000 (0:00:00.064) 0:00:57.312 ***** 2021-02-11T00:23:45.914 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-11T00:23:45.915 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:45.961 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:45 +0000 (0:00:00.075) 0:00:57.388 ***** 2021-02-11T00:23:45.992 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-11T00:23:45.993 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:46.029 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:46 +0000 (0:00:00.067) 0:00:57.455 ***** 2021-02-11T00:23:46.056 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:46.091 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:46 +0000 (0:00:00.062) 0:00:57.518 ***** 2021-02-11T00:23:46.118 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-11T00:23:46.120 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:46.153 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:46 +0000 (0:00:00.061) 0:00:57.580 ***** 2021-02-11T00:23:46.179 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-11T00:23:46.181 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:46.214 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:46 +0000 (0:00:00.060) 0:00:57.641 ***** 2021-02-11T00:23:46.241 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-02-11T00:23:46.242 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:46.276 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:46 +0000 (0:00:00.061) 0:00:57.702 ***** 2021-02-11T00:23:46.302 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-02-11T00:23:46.304 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T00:23:46.337 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:46 +0000 (0:00:00.060) 0:00:57.763 ***** 2021-02-11T00:23:46.363 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-02-11T00:23:46.364 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:46.400 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:46 +0000 (0:00:00.063) 0:00:57.827 ***** 2021-02-11T00:23:46.427 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-11T00:23:46.429 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:46.464 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:46 +0000 (0:00:00.063) 0:00:57.890 ***** 2021-02-11T00:23:46.490 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-02-11T00:23:46.492 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:46.525 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:46 +0000 (0:00:00.061) 0:00:57.952 ***** 2021-02-11T00:23:46.564 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-02-11T00:23:46.566 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:46.576 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:46.615 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:46 +0000 (0:00:00.089) 0:00:58.041 ***** 2021-02-11T00:23:46.654 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-11T00:23:46.655 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi057.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-02-11T00:23:46.692 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:46 +0000 (0:00:00.077) 0:00:58.119 ***** 2021-02-11T00:23:46.720 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-11T00:23:46.721 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:46.756 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:46 +0000 (0:00:00.064) 0:00:58.183 ***** 2021-02-11T00:23:46.906 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-11T00:23:46.908 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.013918", "end": "2021-02-11 00:23:47.096041", "rc": 0, "start": "2021-02-11 00:23:47.082123", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-02-11T00:23:46.943 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:46 +0000 (0:00:00.186) 0:00:58.369 ***** 2021-02-11T00:23:46.971 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-02-11T00:23:46.972 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:47.006 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:46 +0000 (0:00:00.063) 0:00:58.433 ***** 2021-02-11T00:23:47.033 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:47.068 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:47 +0000 (0:00:00.062) 0:00:58.495 ***** 2021-02-11T00:23:47.106 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-02-11T00:23:47.108 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-02-11T00:23:47.118 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-02-11T00:23:47.119 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-02-11T00:23:47.131 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-11T00:23:47.168 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:47 +0000 (0:00:00.099) 0:00:58.594 ***** 2021-02-11T00:23:47.194 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-11T00:23:47.196 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:47.230 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:47 +0000 (0:00:00.061) 0:00:58.656 ***** 2021-02-11T00:23:47.257 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:47.293 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:47 +0000 (0:00:00.063) 0:00:58.720 ***** 2021-02-11T00:23:47.320 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-02-11T00:23:47.322 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:47.358 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:47 +0000 (0:00:00.065) 0:00:58.785 ***** 2021-02-11T00:23:47.386 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-02-11T00:23:47.387 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:47.423 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:47 +0000 (0:00:00.064) 0:00:58.849 ***** 2021-02-11T00:23:47.449 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:47.488 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:47 +0000 (0:00:00.064) 0:00:58.914 ***** 2021-02-11T00:23:48.314 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-02-11T00:23:48.316 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.698062", "end": "2021-02-11 00:23:48.500333", "rc": 0, "start": "2021-02-11 00:23:47.802271", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:23:48.355 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:48 +0000 (0:00:00.867) 0:00:59.782 ***** 2021-02-11T00:23:48.382 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-02-11T00:23:48.383 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:48.421 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:48 +0000 (0:00:00.065) 0:00:59.847 ***** 2021-02-11T00:23:48.597 INFO:teuthology.task.ansible.out: 2021-02-11T00:23:48.597 INFO:teuthology.task.ansible.out:TASK [common : Clean apt cache] ************************************************ 2021-02-11T00:23:48.599 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.029291", "end": "2021-02-11 00:23:48.786630", "rc": 0, "start": "2021-02-11 00:23:48.757339", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:23:48.634 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:48 +0000 (0:00:00.213) 0:01:00.060 ***** 2021-02-11T00:23:48.796 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-02-11T00:23:48.798 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1613002785, "cache_updated": false, "changed": false} 2021-02-11T00:23:48.833 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:48 +0000 (0:00:00.198) 0:01:00.259 ***** 2021-02-11T00:23:49.700 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-02-11T00:23:49.702 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"cache_update_time": 1613002785, "cache_updated": false, "changed": false} 2021-02-11T00:23:49.736 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:49 +0000 (0:00:00.903) 0:01:01.162 ***** 2021-02-11T00:23:49.762 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-02-11T00:23:49.763 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:49.797 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:49 +0000 (0:00:00.061) 0:01:01.223 ***** 2021-02-11T00:23:50.387 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-02-11T00:23:50.388 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-02-11T00:23:50.422 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:50 +0000 (0:00:00.625) 0:01:01.849 ***** 2021-02-11T00:23:51.276 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-02-11T00:23:51.277 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"cache_update_time": 1613002785, "cache_updated": false, "changed": false} 2021-02-11T00:23:51.311 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:51 +0000 (0:00:00.888) 0:01:02.738 ***** 2021-02-11T00:23:51.649 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-11T00:23:52.849 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} ok: [smithi095.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} ok: [smithi095.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} Thursday 11 February 2021 00:23:52 +0000 (0:00:00.941) 0:01:03.679 ***** TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} Thursday 11 February 2021 00:23:52 +0000 (0:00:00.155) 0:01:03.834 ***** TASK [common : Update apt cache] *********************************************** ok: [smithi057.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1613002741, "cache_updated": false, "changed": false} Thursday 11 February 2021 00:23:52 +0000 (0:00:00.053) 0:01:03.888 ***** TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi095.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a4 2021-02-11T00:23:52.851 INFO:teuthology.task.ansible.out:8f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-02-11T00:23:52.996 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-02-11T00:23:53.270 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-02-11T00:23:53.308 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:53 +0000 (0:00:00.846) 0:01:04.735 ***** 2021-02-11T00:23:53.347 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** ok: [smithi095.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-02-11T00:23:53.384 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:53 +0000 (0:00:00.075) 0:01:04.810 ***** 2021-02-11T00:23:53.390 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-02-11T00:23:53.391 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"cache_update_time": 1613002741, "cache_updated": false, "changed": false} 2021-02-11T00:23:53.440 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:53 +0000 (0:00:00.055) 0:01:04.866 ***** 2021-02-11T00:23:53.447 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-02-11T00:23:53.448 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:53.485 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:53 +0000 (0:00:00.045) 0:01:04.911 ***** 2021-02-11T00:23:53.492 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-02-11T00:23:53.493 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:53.530 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:53 +0000 (0:00:00.045) 0:01:04.956 ***** 2021-02-11T00:23:53.538 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-02-11T00:23:53.539 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T00:23:53.578 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:53 +0000 (0:00:00.048) 0:01:05.005 ***** 2021-02-11T00:23:53.619 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-02-11T00:23:53.620 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T00:23:53.630 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T00:23:53.631 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T00:23:53.642 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T00:23:53.687 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:53 +0000 (0:00:00.108) 0:01:05.114 ***** 2021-02-11T00:23:53.714 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-02-11T00:23:53.715 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:53.755 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:53 +0000 (0:00:00.068) 0:01:05.182 ***** 2021-02-11T00:23:53.854 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-02-11T00:23:53.856 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-02-11T00:23:53.891 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:53 +0000 (0:00:00.135) 0:01:05.318 ***** 2021-02-11T00:23:54.631 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-02-11T00:23:54.632 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"cache_update_time": 1613002785, "cache_updated": false, "changed": false} 2021-02-11T00:23:54.668 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:54 +0000 (0:00:00.776) 0:01:06.094 ***** 2021-02-11T00:23:54.788 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-02-11T00:23:54.789 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"cache_update_time": 1613002741, "cache_updated": false, "changed": false} 2021-02-11T00:23:54.827 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:54 +0000 (0:00:00.159) 0:01:06.253 ***** 2021-02-11T00:23:54.948 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-02-11T00:23:54.950 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-02-11T00:23:54.985 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:54 +0000 (0:00:00.158) 0:01:06.412 ***** 2021-02-11T00:23:55.143 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi057.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-02-11T00:23:55.278 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-02-11T00:23:55.280 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-02-11T00:23:55.313 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:55 +0000 (0:00:00.328) 0:01:06.740 ***** 2021-02-11T00:23:55.433 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-11T00:23:55.435 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-02-11T00:23:55.455 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-11T00:23:55.456 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-02-11T00:23:55.560 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-02-11T00:23:55.597 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:55 +0000 (0:00:00.283) 0:01:07.024 ***** 2021-02-11T00:23:55.624 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-02-11T00:23:55.625 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:55.661 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:55 +0000 (0:00:00.064) 0:01:07.088 ***** 2021-02-11T00:23:55.688 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** skipping: [smithi095.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T00:23:55.724 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:55 +0000 (0:00:00.063) 0:01:07.151 ***** 2021-02-11T00:23:55.731 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-11T00:23:55.731 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-02-11T00:23:55.769 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:55 +0000 (0:00:00.044) 0:01:07.196 ***** 2021-02-11T00:23:55.777 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-02-11T00:23:55.779 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:55.815 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:55 +0000 (0:00:00.045) 0:01:07.242 ***** 2021-02-11T00:23:55.894 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-02-11T00:23:55.896 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T00:23:55.931 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:55 +0000 (0:00:00.116) 0:01:07.358 ***** 2021-02-11T00:23:56.128 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-02-11T00:23:56.130 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-02-11T00:23:56.171 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:56 +0000 (0:00:00.239) 0:01:07.598 ***** 2021-02-11T00:23:56.241 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-11T00:23:56.241 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-02-11T00:23:56.516 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-02-11T00:23:56.767 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-02-11T00:23:56.805 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:56 +0000 (0:00:00.633) 0:01:08.231 ***** 2021-02-11T00:23:56.843 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-02-11T00:23:56.844 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-02-11T00:23:56.886 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:56 +0000 (0:00:00.081) 0:01:08.313 ***** 2021-02-11T00:23:56.893 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-02-11T00:23:56.895 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 00:22:17 UTC", "ActiveEnterTimestampMonotonic": "21362335", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket var-run.mount system.slice -.mount basic.target remote-fs.target local-fs.target nss-lookup.target time-sync.target network.target sysinit.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 00:22:17 UTC", "AssertTimestampMonotonic": "21359689", "Before": "shutdown.target multi-user.target getty@tty1.service xdm.service plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 00:22:17 UTC", "ConditionTimestampMonotonic": "21359688", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-02-11T00:23:56.897 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1149", "ExecMainStartTimestamp": "Thu 2021-02-11 00:22:17 UTC", "ExecMainStartTimestampMonotonic": "21362233", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2021-02-11 00:22:17 UTC] ; stop_time=[n/a] ; pid=1149 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 00:22:17 UTC", "InactiveExitTimestampMonotonic": "21362335", "InvocationID": "7082e901267e425faa44ef86a7269276", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2021-02-11T00:23:56.898 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1149", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2021-02-11T00:23:56.899 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 00:22:17 UTC", "StateChangeTimestampMonotonic": "21362335", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 00:22:17 UTC", "WatchdogTimestampMonotonic": "21362332", "WatchdogUSec": "0"}} 2021-02-11T00:23:56.933 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:56 +0000 (0:00:00.046) 0:01:08.360 ***** 2021-02-11T00:23:56.940 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-02-11T00:23:56.941 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:56.977 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:56 +0000 (0:00:00.043) 0:01:08.403 ***** 2021-02-11T00:23:56.983 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-02-11T00:23:56.985 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:57.020 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:57 +0000 (0:00:00.043) 0:01:08.447 ***** 2021-02-11T00:23:57.030 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-02-11T00:23:57.031 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T00:23:57.069 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:57 +0000 (0:00:00.048) 0:01:08.496 ***** 2021-02-11T00:23:57.076 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-02-11T00:23:57.077 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:57.114 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:57 +0000 (0:00:00.044) 0:01:08.540 ***** 2021-02-11T00:23:57.121 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-02-11T00:23:57.123 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi057.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi057.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T00:23:57.124 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-11T00:23:57.194 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:57 +0000 (0:00:00.080) 0:01:08.621 ***** 2021-02-11T00:23:57.201 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-02-11T00:23:57.202 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:57.238 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:57 +0000 (0:00:00.043) 0:01:08.664 ***** 2021-02-11T00:23:57.244 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-02-11T00:23:57.245 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:57.281 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:57 +0000 (0:00:00.042) 0:01:08.707 ***** 2021-02-11T00:23:57.286 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-02-11T00:23:57.288 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:57.324 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:57 +0000 (0:00:00.042) 0:01:08.750 ***** 2021-02-11T00:23:57.351 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:57.387 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:57 +0000 (0:00:00.062) 0:01:08.813 ***** 2021-02-11T00:23:57.414 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-02-11T00:23:57.415 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:57.451 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:57 +0000 (0:00:00.063) 0:01:08.877 ***** 2021-02-11T00:23:57.478 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-02-11T00:23:57.479 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:57.512 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:57 +0000 (0:00:00.061) 0:01:08.939 ***** 2021-02-11T00:23:57.539 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-02-11T00:23:57.540 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:57.574 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:57 +0000 (0:00:00.061) 0:01:09.000 ***** 2021-02-11T00:23:57.606 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-02-11T00:23:57.607 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:57.647 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:57 +0000 (0:00:00.072) 0:01:09.073 ***** 2021-02-11T00:23:57.674 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-02-11T00:23:57.676 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:57.709 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:57 +0000 (0:00:00.062) 0:01:09.136 ***** 2021-02-11T00:23:57.737 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-02-11T00:23:57.738 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-02-11T00:23:57.776 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:57 +0000 (0:00:00.066) 0:01:09.202 ***** 2021-02-11T00:23:57.814 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-02-11T00:23:57.815 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-02-11T00:23:57.853 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:57 +0000 (0:00:00.077) 0:01:09.280 ***** 2021-02-11T00:23:57.891 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-02-11T00:23:57.892 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-02-11T00:23:57.928 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:57 +0000 (0:00:00.074) 0:01:09.354 ***** 2021-02-11T00:23:59.278 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-02-11T00:23:59.279 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"cache_update_time": 1613002741, "cache_updated": false, "changed": false} 2021-02-11T00:23:59.358 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:59 +0000 (0:00:01.430) 0:01:10.785 ***** 2021-02-11T00:23:59.365 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-02-11T00:23:59.366 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-11T00:23:59.403 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:59 +0000 (0:00:00.045) 0:01:10.830 ***** 2021-02-11T00:23:59.566 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-02-11T00:23:59.567 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-02-11T00:23:59.603 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:59 +0000 (0:00:00.199) 0:01:11.029 ***** 2021-02-11T00:23:59.724 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-02-11T00:23:59.726 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-02-11T00:23:59.763 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:59 +0000 (0:00:00.160) 0:01:11.189 ***** 2021-02-11T00:23:59.770 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-02-11T00:23:59.770 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-02-11T00:23:59.809 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:59 +0000 (0:00:00.045) 0:01:11.235 ***** 2021-02-11T00:23:59.835 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-02-11T00:23:59.837 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:59.874 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:59 +0000 (0:00:00.064) 0:01:11.300 ***** 2021-02-11T00:23:59.901 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-02-11T00:23:59.903 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:23:59.939 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:23:59 +0000 (0:00:00.065) 0:01:11.365 ***** 2021-02-11T00:24:00.067 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-02-11T00:24:00.068 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-02-11T00:24:00.107 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:00 +0000 (0:00:00.167) 0:01:11.533 ***** 2021-02-11T00:24:00.113 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-02-11T00:24:00.114 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-02-11T00:24:00.153 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:00 +0000 (0:00:00.046) 0:01:11.580 ***** 2021-02-11T00:24:00.265 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-11T00:24:00.267 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-02-11T00:24:00.371 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-02-11T00:24:00.411 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:00 +0000 (0:00:00.257) 0:01:11.837 ***** 2021-02-11T00:24:00.439 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-02-11T00:24:00.440 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:00.476 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:00 +0000 (0:00:00.064) 0:01:11.902 ***** 2021-02-11T00:24:00.505 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-02-11T00:24:00.506 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-11T00:24:00.550 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:00 +0000 (0:00:00.074) 0:01:11.977 ***** 2021-02-11T00:24:00.569 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-02-11T00:24:00.569 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-02-11T00:24:00.611 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:00 +0000 (0:00:00.060) 0:01:12.037 ***** 2021-02-11T00:24:00.620 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-02-11T00:24:00.620 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:00.663 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:00 +0000 (0:00:00.052) 0:01:12.090 ***** 2021-02-11T00:24:00.672 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-02-11T00:24:00.673 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi095"}, "changed": false} 2021-02-11T00:24:00.711 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:00 +0000 (0:00:00.047) 0:01:12.137 ***** 2021-02-11T00:24:00.991 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-02-11T00:24:00.991 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-02-11T00:24:01.052 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:01 +0000 (0:00:00.341) 0:01:12.479 ***** 2021-02-11T00:24:01.269 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-02-11T00:24:01.270 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 00:22:12 UTC", "ActiveEnterTimestampMonotonic": "18798682", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice var-run.mount nss-lookup.target systemd-tmpfiles-setup.service -.mount systemd-journald.socket local-fs.target network.target sysinit.target remote-fs.target time-sync.target basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 00:22:12 UTC", "AssertTimestampMonotonic": "18797972", "Before": "xdm.service getty@tty1.service plymouth-quit.service multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 00:22:12 UTC", "ConditionTimestampMonotonic": "18797972", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-02-11T00:24:01.272 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1153", "ExecMainStartTimestamp": "Thu 2021-02-11 00:22:12 UTC", "ExecMainStartTimestampMonotonic": "18798651", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2021-02-11 00:22:12 UTC] ; stop_time=[n/a] ; pid=1153 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 00:22:12 UTC", "InactiveExitTimestampMonotonic": "18798682", "InvocationID": "8f83ab6a61314d8fbf1543062b8ad3e7", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2021-02-11T00:24:01.273 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1153", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2021-02-11T00:24:01.274 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 00:22:12 UTC", "StateChangeTimestampMonotonic": "18798682", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 00:22:12 UTC", "WatchdogTimestampMonotonic": "18798682", "WatchdogUSec": "0"}} 2021-02-11T00:24:01.322 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:01 +0000 (0:00:00.270) 0:01:12.749 ***** 2021-02-11T00:24:01.351 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-02-11T00:24:01.352 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:01.395 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:01 +0000 (0:00:00.072) 0:01:12.821 ***** 2021-02-11T00:24:01.423 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-02-11T00:24:01.425 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:01.468 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:01 +0000 (0:00:00.073) 0:01:12.895 ***** 2021-02-11T00:24:01.496 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-02-11T00:24:01.497 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:01.532 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:01 +0000 (0:00:00.063) 0:01:12.958 ***** 2021-02-11T00:24:01.559 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-02-11T00:24:01.560 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:01.595 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:01 +0000 (0:00:00.063) 0:01:13.022 ***** 2021-02-11T00:24:01.623 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-02-11T00:24:01.624 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:01.659 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:01 +0000 (0:00:00.063) 0:01:13.086 ***** 2021-02-11T00:24:01.696 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-02-11T00:24:01.698 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:01.733 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:01 +0000 (0:00:00.073) 0:01:13.159 ***** 2021-02-11T00:24:01.760 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-02-11T00:24:01.762 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi095.front.sepia.ceph.com", "ansible_hostname": "smithi095", "ansible_nodename": "smithi095"}, "changed": true, "name": "smithi095"} TASK [common : nrpe - Compile SELinux module file] ***************************** skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:01.798 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:01 +0000 (0:00:00.064) 0:01:13.224 ***** 2021-02-11T00:24:01.840 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:01 +0000 (0:00:00.042) 0:01:13.266 ***** 2021-02-11T00:24:01.845 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-02-11T00:24:01.847 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:01.888 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:01 +0000 (0:00:00.047) 0:01:13.314 ***** 2021-02-11T00:24:01.914 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-02-11T00:24:01.915 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:01.950 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:01 +0000 (0:00:00.062) 0:01:13.376 ***** 2021-02-11T00:24:01.977 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-02-11T00:24:01.978 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:02.012 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:02 +0000 (0:00:00.062) 0:01:13.439 ***** 2021-02-11T00:24:02.050 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-02-11T00:24:02.051 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-02-11T00:24:02.087 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:02 +0000 (0:00:00.074) 0:01:13.513 ***** 2021-02-11T00:24:02.135 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-02-11T00:24:02.136 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-02-11T00:24:02.172 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:02 +0000 (0:00:00.084) 0:01:13.598 ***** 2021-02-11T00:24:02.186 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-02-11T00:24:02.186 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-02-11T00:24:02.223 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:02 +0000 (0:00:00.051) 0:01:13.649 ***** 2021-02-11T00:24:02.346 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-02-11T00:24:02.347 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-02-11T00:24:02.478 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:02 +0000 (0:00:00.255) 0:01:13.905 ***** 2021-02-11T00:24:02.516 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-02-11T00:24:02.517 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-11T00:24:02.552 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:02 +0000 (0:00:00.073) 0:01:13.978 ***** 2021-02-11T00:24:02.674 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-02-11T00:24:02.675 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-02-11T00:24:02.677 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-02-11T00:24:02.710 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:02 +0000 (0:00:00.158) 0:01:14.137 ***** 2021-02-11T00:24:02.743 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:02 +0000 (0:00:00.032) 0:01:14.170 ***** 2021-02-11T00:24:02.843 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-02-11T00:24:02.844 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-02-11T00:24:02.880 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:02 +0000 (0:00:00.137) 0:01:14.307 ***** 2021-02-11T00:24:02.909 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-02-11T00:24:02.910 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:02.942 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:02 +0000 (0:00:00.062) 0:01:14.369 ***** 2021-02-11T00:24:02.970 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:03.012 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:03 +0000 (0:00:00.069) 0:01:14.438 ***** 2021-02-11T00:24:03.164 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-02-11T00:24:03.164 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-02-11T00:24:03.201 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:03 +0000 (0:00:00.189) 0:01:14.628 ***** 2021-02-11T00:24:03.281 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** changed: [smithi095.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-02-11T00:24:03.406 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:03 +0000 (0:00:00.118) 0:01:14.746 ***** TASK [testnode : Create remote.conf] ******************************************* skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 00:24:03 +0000 (0:00:00.073) 0:01:14.819 ***** 2021-02-11T00:24:03.419 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-11T00:24:03.420 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:03.459 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:03 +0000 (0:00:00.066) 0:01:14.886 ***** 2021-02-11T00:24:03.496 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:03.534 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:03 +0000 (0:00:00.074) 0:01:14.960 ***** 2021-02-11T00:24:03.560 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:03.595 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:03 +0000 (0:00:00.060) 0:01:15.021 ***** 2021-02-11T00:24:03.600 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-02-11T00:24:03.601 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-02-11T00:24:03.637 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:03 +0000 (0:00:00.042) 0:01:15.064 ***** 2021-02-11T00:24:03.644 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-11T00:24:03.646 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:03.680 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:03 +0000 (0:00:00.042) 0:01:15.106 ***** 2021-02-11T00:24:03.690 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-02-11T00:24:03.691 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi057"}, "changed": false} 2021-02-11T00:24:03.726 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:03 +0000 (0:00:00.045) 0:01:15.152 ***** 2021-02-11T00:24:03.731 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-11T00:24:03.731 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:03.767 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:03 +0000 (0:00:00.041) 0:01:15.194 ***** 2021-02-11T00:24:03.794 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-11T00:24:03.795 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:03.829 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:03 +0000 (0:00:00.061) 0:01:15.256 ***** 2021-02-11T00:24:03.855 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-02-11T00:24:03.857 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:03.890 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:03 +0000 (0:00:00.061) 0:01:15.317 ***** 2021-02-11T00:24:03.916 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-11T00:24:03.949 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:03 +0000 (0:00:00.058) 0:01:15.376 ***** 2021-02-11T00:24:03.975 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-02-11T00:24:04.011 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:04 +0000 (0:00:00.061) 0:01:15.437 ***** 2021-02-11T00:24:04.038 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-02-11T00:24:04.077 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:04 +0000 (0:00:00.066) 0:01:15.504 ***** 2021-02-11T00:24:04.106 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-02-11T00:24:04.107 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:04.142 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:04 +0000 (0:00:00.064) 0:01:15.569 ***** 2021-02-11T00:24:04.169 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-02-11T00:24:04.171 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi057.front.sepia.ceph.com", "ansible_hostname": "smithi057", "ansible_nodename": "smithi057"}, "changed": true, "name": "smithi057"} 2021-02-11T00:24:04.205 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:04 +0000 (0:00:00.063) 0:01:15.632 ***** 2021-02-11T00:24:04.212 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-02-11T00:24:04.212 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:04.213 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:04.250 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:04 +0000 (0:00:00.044) 0:01:15.677 ***** 2021-02-11T00:24:04.286 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-02-11T00:24:04.287 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:04.319 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:04 +0000 (0:00:00.069) 0:01:15.746 ***** 2021-02-11T00:24:04.346 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T00:24:04.347 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:04.381 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:04 +0000 (0:00:00.061) 0:01:15.807 ***** 2021-02-11T00:24:04.407 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:04.442 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:04 +0000 (0:00:00.061) 0:01:15.868 ***** 2021-02-11T00:24:04.468 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-02-11T00:24:04.470 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:04.504 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:04 +0000 (0:00:00.062) 0:01:15.931 ***** 2021-02-11T00:24:04.521 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-02-11T00:24:04.521 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-02-11T00:24:04.555 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:04 +0000 (0:00:00.051) 0:01:15.982 ***** 2021-02-11T00:24:04.562 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-02-11T00:24:04.563 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:04.601 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:04 +0000 (0:00:00.045) 0:01:16.027 ***** 2021-02-11T00:24:04.627 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-02-11T00:24:04.629 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:04.662 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:04 +0000 (0:00:00.061) 0:01:16.089 ***** 2021-02-11T00:24:04.689 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-11T00:24:04.765 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 00:24:04 +0000 (0:00:00.075) 0:01:16.165 ***** TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:04.800 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:04 +0000 (0:00:00.061) 0:01:16.226 ***** 2021-02-11T00:24:04.827 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:04.867 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:04 +0000 (0:00:00.067) 0:01:16.294 ***** 2021-02-11T00:24:04.873 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-02-11T00:24:04.874 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-02-11T00:24:04.908 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:04 +0000 (0:00:00.041) 0:01:16.335 ***** 2021-02-11T00:24:04.914 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-02-11T00:24:04.915 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:04.950 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:04 +0000 (0:00:00.041) 0:01:16.376 ***** 2021-02-11T00:24:04.976 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-11T00:24:04.978 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:05.016 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:05 +0000 (0:00:00.066) 0:01:16.443 ***** 2021-02-11T00:24:05.043 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T00:24:05.044 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:05.082 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:05 +0000 (0:00:00.065) 0:01:16.508 ***** 2021-02-11T00:24:05.108 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-02-11T00:24:05.109 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:05.141 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:05 +0000 (0:00:00.059) 0:01:16.568 ***** 2021-02-11T00:24:05.166 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-02-11T00:24:05.167 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:05.201 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:05 +0000 (0:00:00.060) 0:01:16.628 ***** 2021-02-11T00:24:05.228 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-02-11T00:24:05.229 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:05.260 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:05 +0000 (0:00:00.058) 0:01:16.687 ***** 2021-02-11T00:24:05.285 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-02-11T00:24:05.287 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:05.320 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:05 +0000 (0:00:00.060) 0:01:16.747 ***** 2021-02-11T00:24:05.347 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-02-11T00:24:05.349 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:05.387 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:05 +0000 (0:00:00.066) 0:01:16.814 ***** 2021-02-11T00:24:05.629 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 00:24:05 +0000 (0:00:00.062) 0:01:16.876 ***** TASK [testnode : Add ssh pubkeys] ********************************************** changed: [smithi057.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} Thursday 11 February 2021 00:24:05 +0000 (0:00:00.040) 0:01:16.916 ***** TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 00:24:05 +0000 (0:00:00.039) 0:01:16.956 ***** TASK [testnode : Create remote.conf] ******************************************* skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 00:24:05 +0000 (0:00:00.038) 0:01:16.994 ***** TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 00:24:05 +0000 (0:00:00.039) 0:01:17.034 ***** TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:05.650 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:05 +0000 (0:00:00.042) 0:01:17.076 ***** 2021-02-11T00:24:05.677 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-11T00:24:05.678 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:05.720 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:05 +0000 (0:00:00.070) 0:01:17.147 ***** 2021-02-11T00:24:05.748 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-02-11T00:24:05.749 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.004303", "end": "2021-02-11 00:24:05.733470", "rc": 0, "start": "2021-02-11 00:24:05.729167", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:05.781 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:05 +0000 (0:00:00.060) 0:01:17.207 ***** 2021-02-11T00:24:05.814 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:05 +0000 (0:00:00.032) 0:01:17.240 ***** 2021-02-11T00:24:05.820 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-11T00:24:05.821 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:05.855 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:05 +0000 (0:00:00.041) 0:01:17.281 ***** 2021-02-11T00:24:05.861 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-02-11T00:24:05.898 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:05 +0000 (0:00:00.043) 0:01:17.324 ***** 2021-02-11T00:24:05.903 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-11T00:24:05.905 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:05.940 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:05 +0000 (0:00:00.041) 0:01:17.366 ***** 2021-02-11T00:24:05.967 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-11T00:24:05.968 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:06.000 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:05 +0000 (0:00:00.059) 0:01:17.426 ***** 2021-02-11T00:24:06.026 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-02-11T00:24:06.027 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:06.079 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:06 +0000 (0:00:00.064) 0:01:17.491 ***** 2021-02-11T00:24:06.094 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-11T00:24:06.129 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:06 +0000 (0:00:00.064) 0:01:17.555 ***** 2021-02-11T00:24:06.155 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-02-11T00:24:06.190 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:06 +0000 (0:00:00.061) 0:01:17.616 ***** 2021-02-11T00:24:06.207 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-02-11T00:24:06.207 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-02-11T00:24:06.242 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:06 +0000 (0:00:00.052) 0:01:17.669 ***** 2021-02-11T00:24:06.248 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-02-11T00:24:06.284 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:06 +0000 (0:00:00.041) 0:01:17.711 ***** 2021-02-11T00:24:06.290 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-02-11T00:24:06.290 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:06.324 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:06 +0000 (0:00:00.040) 0:01:17.751 ***** 2021-02-11T00:24:06.331 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-02-11T00:24:06.332 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:06.368 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:06 +0000 (0:00:00.043) 0:01:17.794 ***** 2021-02-11T00:24:06.426 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-02-11T00:24:06.427 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi057.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:06.464 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:06 +0000 (0:00:00.095) 0:01:17.890 ***** 2021-02-11T00:24:06.491 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:06.525 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:06 +0000 (0:00:00.061) 0:01:17.951 ***** 2021-02-11T00:24:06.552 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:06.584 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:06 +0000 (0:00:00.059) 0:01:18.011 ***** 2021-02-11T00:24:06.611 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:06.644 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:06 +0000 (0:00:00.059) 0:01:18.070 ***** 2021-02-11T00:24:06.670 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-02-11T00:24:06.671 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:06.755 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:06 +0000 (0:00:00.061) 0:01:18.132 ***** TASK [testnode : Check if ceph-debuginfo is installed] ************************* skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:06.768 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:06 +0000 (0:00:00.062) 0:01:18.195 ***** 2021-02-11T00:24:06.795 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-02-11T00:24:06.796 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:06.830 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:06 +0000 (0:00:00.061) 0:01:18.257 ***** 2021-02-11T00:24:06.856 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-11T00:24:06.858 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:06.889 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:06 +0000 (0:00:00.058) 0:01:18.315 ***** 2021-02-11T00:24:06.915 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:06.949 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:06 +0000 (0:00:00.060) 0:01:18.376 ***** 2021-02-11T00:24:06.975 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T00:24:06.977 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:07.010 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:07 +0000 (0:00:00.061) 0:01:18.437 ***** 2021-02-11T00:24:07.037 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-02-11T00:24:07.038 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:07.070 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:07 +0000 (0:00:00.060) 0:01:18.497 ***** 2021-02-11T00:24:07.097 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-11T00:24:07.099 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:07.131 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:07 +0000 (0:00:00.060) 0:01:18.558 ***** 2021-02-11T00:24:07.157 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T00:24:07.158 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:07.194 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:07 +0000 (0:00:00.062) 0:01:18.621 ***** 2021-02-11T00:24:07.220 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-02-11T00:24:07.221 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:07.255 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:07 +0000 (0:00:00.060) 0:01:18.681 ***** 2021-02-11T00:24:07.334 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 00:24:07 +0000 (0:00:00.063) 0:01:18.745 ***** 2021-02-11T00:24:07.345 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-02-11T00:24:07.346 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:07.380 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:07 +0000 (0:00:00.061) 0:01:18.806 ***** 2021-02-11T00:24:07.406 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-02-11T00:24:07.407 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:07.440 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:07 +0000 (0:00:00.060) 0:01:18.867 ***** 2021-02-11T00:24:07.466 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-02-11T00:24:07.468 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:07.500 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:07 +0000 (0:00:00.059) 0:01:18.927 ***** 2021-02-11T00:24:07.526 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-02-11T00:24:07.527 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:07.561 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:07 +0000 (0:00:00.060) 0:01:18.987 ***** 2021-02-11T00:24:07.588 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-02-11T00:24:07.590 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:07.624 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:07 +0000 (0:00:00.063) 0:01:19.051 ***** 2021-02-11T00:24:07.651 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-02-11T00:24:07.652 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:07.685 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:07 +0000 (0:00:00.060) 0:01:19.112 ***** 2021-02-11T00:24:07.817 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-02-11T00:24:07.818 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-02-11T00:24:07.838 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-02-11T00:24:07.840 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.003503", "end": "2021-02-11 00:24:08.014590", "rc": 0, "start": "2021-02-11 00:24:08.011087", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:07.888 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:07 +0000 (0:00:00.202) 0:01:19.314 ***** 2021-02-11T00:24:08.972 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** Thursday 11 February 2021 00:24:07 +0000 (0:00:00.065) 0:01:19.380 ***** TASK [testnode : Set apt preferences] ****************************************** ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} Thursday 11 February 2021 00:24:08 +0000 (0:00:00.343) 0:01:19.724 ***** TASK [testnode : Add sources list] ********************************************* skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 00:24:08 +0000 (0:00:00.060) 0:01:19.785 ***** TASK [testnode : Install apt keys] ********************************************* ok: [smithi095.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} Thursday 11 February 2021 00:24:08 +0000 (0:00:00.419) 0:01:20.204 ***** 2021-02-11T00:24:09.368 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-02-11T00:24:09.679 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-02-11T00:24:09.681 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"cache_update_time": 1613002785, "cache_updated": false, "changed": false} 2021-02-11T00:24:09.715 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:09 +0000 (0:00:00.937) 0:01:21.141 ***** 2021-02-11T00:24:09.742 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-02-11T00:24:09.777 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:09 +0000 (0:00:00.061) 0:01:21.203 ***** 2021-02-11T00:24:09.949 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-02-11T00:24:09.950 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.036235", "end": "2021-02-11 00:24:09.932197", "rc": 0, "start": "2021-02-11 00:24:09.895962", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:09.984 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:09 +0000 (0:00:00.207) 0:01:21.411 ***** 2021-02-11T00:24:10.314 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-02-11T00:24:10.316 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-02-11T00:24:10.350 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:10 +0000 (0:00:00.365) 0:01:21.777 ***** 2021-02-11T00:24:11.255 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-02-11T00:24:11.256 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"cache_update_time": 1613002741, "cache_updated": false, "changed": false} 2021-02-11T00:24:11.289 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:11 +0000 (0:00:00.938) 0:01:22.716 ***** 2021-02-11T00:24:11.317 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-02-11T00:24:11.353 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:11 +0000 (0:00:00.063) 0:01:22.780 ***** 2021-02-11T00:24:11.526 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-02-11T00:24:11.526 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.037530", "end": "2021-02-11 00:24:11.699661", "rc": 0, "start": "2021-02-11 00:24:11.662131", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:11.561 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:11 +0000 (0:00:00.207) 0:01:22.987 ***** 2021-02-11T00:24:12.988 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-02-11T00:24:12.990 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"cache_update_time": 1613002785, "cache_updated": false, "changed": false} 2021-02-11T00:24:13.027 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:13 +0000 (0:00:01.465) 0:01:24.453 ***** 2021-02-11T00:24:14.719 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"cache_update_time": 1613002741, "cache_updated": false, "changed": false} 2021-02-11T00:24:14.876 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:14 +0000 (0:00:01.730) 0:01:26.183 ***** 2021-02-11T00:24:22.305 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-02-11T00:24:22.305 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 19 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137610 files and directorie 2021-02-11T00:24:22.306 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 19 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-02-11T00:24:22.308 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137610 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-02-11T00:24:22.342 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:22 +0000 (0:00:07.585) 0:01:33.769 ***** 2021-02-11T00:24:23.233 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T00:24:23.235 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"cache_update_time": 1613002785, "cache_updated": false, "changed": false} 2021-02-11T00:24:23.270 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:23 +0000 (0:00:00.927) 0:01:34.696 ***** 2021-02-11T00:24:23.276 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-02-11T00:24:23.277 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 19 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137610 files and directorie 2021-02-11T00:24:23.278 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 19 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-02-11T00:24:23.280 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137610 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.34) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.34) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-02-11T00:24:23.315 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:23 +0000 (0:00:00.045) 0:01:34.742 ***** 2021-02-11T00:24:23.321 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-02-11T00:24:23.322 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:23.358 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:23 +0000 (0:00:00.043) 0:01:34.785 ***** 2021-02-11T00:24:24.263 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T00:24:24.264 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"cache_update_time": 1613002741, "cache_updated": false, "changed": false} 2021-02-11T00:24:24.303 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:24 +0000 (0:00:00.944) 0:01:35.729 ***** 2021-02-11T00:24:24.330 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:24.367 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:24 +0000 (0:00:00.064) 0:01:35.794 ***** 2021-02-11T00:24:31.833 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T00:24:31.834 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"cache_update_time": 1613002785, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 19 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.35 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.35 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.35 [5194 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.35 [868 kB]\nFetched 10.4 MB in 1s (9670 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2021-02-11T00:24:31.835 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137468 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2021-02-11T00:24:31.837 INFO:teuthology.task.ansible.out:u1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 19 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.35 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.35 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.35 [5194 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.35 [868 kB]", "Fetched 10.4 MB in 1s (9670 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2021-02-11T00:24:31.838 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137468 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-02-11T00:24:31.877 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:31 +0000 (0:00:07.509) 0:01:43.304 ***** 2021-02-11T00:24:32.783 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-02-11T00:24:32.784 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"cache_update_time": 1613002785, "cache_updated": false, "changed": false} 2021-02-11T00:24:32.819 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:32 +0000 (0:00:00.941) 0:01:44.245 ***** 2021-02-11T00:24:32.940 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T00:24:32.941 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"cache_update_time": 1613002741, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 19 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.35 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.35 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.35 [5194 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.35 [868 kB]\nFetched 10.4 MB in 1s (9248 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2021-02-11T00:24:32.942 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137468 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2021-02-11T00:24:32.943 INFO:teuthology.task.ansible.out:u1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 19 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.35 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.35 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.35 [5194 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.35 [868 kB]", "Fetched 10.4 MB in 1s (9248 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2021-02-11T00:24:32.945 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137468 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.35_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.35) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.35) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-02-11T00:24:32.977 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:32 +0000 (0:00:00.157) 0:01:44.403 ***** 2021-02-11T00:24:32.983 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-02-11T00:24:32.984 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-02-11T00:24:33.021 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:33 +0000 (0:00:00.044) 0:01:44.448 ***** 2021-02-11T00:24:33.048 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-02-11T00:24:33.049 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:33.083 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:33 +0000 (0:00:00.061) 0:01:44.510 ***** 2021-02-11T00:24:33.223 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-02-11T00:24:33.225 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-02-11T00:24:33.259 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:33 +0000 (0:00:00.175) 0:01:44.686 ***** 2021-02-11T00:24:33.576 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-02-11T00:24:33.577 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-02-11T00:24:33.612 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:33 +0000 (0:00:00.353) 0:01:45.039 ***** 2021-02-11T00:24:33.774 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-02-11T00:24:33.775 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-11T00:24:33.810 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:33 +0000 (0:00:00.197) 0:01:45.236 ***** 2021-02-11T00:24:33.920 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-02-11T00:24:33.922 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"cache_update_time": 1613002741, "cache_updated": false, "changed": false} 2021-02-11T00:24:33.956 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:33 +0000 (0:00:00.146) 0:01:45.383 ***** 2021-02-11T00:24:33.964 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-02-11T00:24:33.965 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T00:24:34.007 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:34 +0000 (0:00:00.050) 0:01:45.433 ***** 2021-02-11T00:24:34.077 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-02-11T00:24:34.078 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-02-11T00:24:34.114 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:34 +0000 (0:00:00.107) 0:01:45.541 ***** 2021-02-11T00:24:34.141 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-02-11T00:24:34.142 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:34.181 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:34 +0000 (0:00:00.066) 0:01:45.607 ***** 2021-02-11T00:24:34.292 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-02-11T00:24:34.293 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 00:22:17 UTC", "ActiveEnterTimestampMonotonic": "21860103", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice systemd-journald.socket network.target nss-lookup.target systemd-tmpfiles-setup.service -.mount basic.target remote-fs.target sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 00:22:17 UTC", "AssertTimestampMonotonic": "21357924", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 00:22:17 UTC", "ConditionTimestampMonotonic": "21357923", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-02-11T00:24:34.294 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1241", "ExecMainStartTimestamp": "Thu 2021-02-11 00:22:17 UTC", "ExecMainStartTimestampMonotonic": "21860090", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 00:22:17 UTC", "InactiveExitTimestampMonotonic": "21359591", "InvocationID": "44a813011f914c02b7e374734822570d", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-02-11T00:24:34.295 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1241", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-02-11T00:24:34.297 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 00:22:17 UTC", "StateChangeTimestampMonotonic": "21860103", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 00:22:17 UTC", "WatchdogTimestampMonotonic": "21860102", "WatchdogUSec": "0"}} 2021-02-11T00:24:34.330 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:34 +0000 (0:00:00.148) 0:01:45.756 ***** 2021-02-11T00:24:34.336 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-02-11T00:24:34.336 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-02-11T00:24:34.373 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:34 +0000 (0:00:00.043) 0:01:45.799 ***** 2021-02-11T00:24:34.379 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-11T00:24:34.380 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:34.416 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:34 +0000 (0:00:00.043) 0:01:45.843 ***** 2021-02-11T00:24:34.443 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:34.479 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:34 +0000 (0:00:00.063) 0:01:45.906 ***** 2021-02-11T00:24:34.506 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-11T00:24:34.508 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:34.541 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:34 +0000 (0:00:00.061) 0:01:45.968 ***** 2021-02-11T00:24:34.569 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-11T00:24:34.570 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:34.609 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:34 +0000 (0:00:00.067) 0:01:46.035 ***** 2021-02-11T00:24:34.636 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-11T00:24:34.637 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:34.673 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:34 +0000 (0:00:00.064) 0:01:46.100 ***** 2021-02-11T00:24:34.681 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-02-11T00:24:34.682 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-02-11T00:24:34.718 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:34 +0000 (0:00:00.044) 0:01:46.145 ***** 2021-02-11T00:24:34.726 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-11T00:24:34.727 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:34.772 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:34 +0000 (0:00:00.054) 0:01:46.199 ***** 2021-02-11T00:24:34.801 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-11T00:24:34.802 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:34.841 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:34 +0000 (0:00:00.068) 0:01:46.268 ***** 2021-02-11T00:24:34.868 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-11T00:24:34.869 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:34.903 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:34 +0000 (0:00:00.061) 0:01:46.329 ***** 2021-02-11T00:24:34.909 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-02-11T00:24:34.909 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-11T00:24:34.954 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:34 +0000 (0:00:00.051) 0:01:46.380 ***** 2021-02-11T00:24:34.961 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T00:24:34.961 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.009 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.054) 0:01:46.435 ***** 2021-02-11T00:24:35.054 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-11T00:24:35.056 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.106 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.097) 0:01:46.533 ***** 2021-02-11T00:24:35.114 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-02-11T00:24:35.115 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T00:24:35.161 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.054) 0:01:46.588 ***** 2021-02-11T00:24:35.167 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T00:24:35.169 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.206 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.044) 0:01:46.633 ***** 2021-02-11T00:24:35.234 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T00:24:35.236 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.270 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.063) 0:01:46.696 ***** 2021-02-11T00:24:35.296 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-11T00:24:35.297 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.332 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.061) 0:01:46.758 ***** 2021-02-11T00:24:35.359 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T00:24:35.360 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.395 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.062) 0:01:46.821 ***** 2021-02-11T00:24:35.422 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.457 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.062) 0:01:46.884 ***** 2021-02-11T00:24:35.464 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-02-11T00:24:35.466 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 00:22:12 UTC", "ActiveEnterTimestampMonotonic": "19242488", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target sysinit.target remote-fs.target system.slice systemd-journald.socket -.mount nss-lookup.target network.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 00:22:12 UTC", "AssertTimestampMonotonic": "18800964", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 00:22:12 UTC", "ConditionTimestampMonotonic": "18800964", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-02-11T00:24:35.467 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1241", "ExecMainStartTimestamp": "Thu 2021-02-11 00:22:12 UTC", "ExecMainStartTimestampMonotonic": "19242476", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 00:22:12 UTC", "InactiveExitTimestampMonotonic": "18802417", "InvocationID": "9fa5cb19866043c8ab0b55975784ab4e", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-02-11T00:24:35.468 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1241", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-02-11T00:24:35.470 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 00:22:12 UTC", "StateChangeTimestampMonotonic": "19242488", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 00:22:12 UTC", "WatchdogTimestampMonotonic": "19242487", "WatchdogUSec": "0"}} 2021-02-11T00:24:35.503 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.045) 0:01:46.929 ***** 2021-02-11T00:24:35.509 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T00:24:35.510 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.545 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.042) 0:01:46.972 ***** 2021-02-11T00:24:35.550 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-11T00:24:35.552 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.587 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.041) 0:01:47.014 ***** 2021-02-11T00:24:35.594 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-11T00:24:35.595 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.629 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.041) 0:01:47.055 ***** 2021-02-11T00:24:35.635 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-11T00:24:35.637 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.670 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.041) 0:01:47.097 ***** 2021-02-11T00:24:35.676 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-11T00:24:35.677 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.713 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.042) 0:01:47.139 ***** 2021-02-11T00:24:35.719 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-11T00:24:35.720 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.758 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.045) 0:01:47.185 ***** 2021-02-11T00:24:35.764 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-11T00:24:35.765 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.798 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.040) 0:01:47.225 ***** 2021-02-11T00:24:35.805 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-11T00:24:35.806 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.840 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.041) 0:01:47.267 ***** 2021-02-11T00:24:35.847 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T00:24:35.848 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.883 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.042) 0:01:47.309 ***** 2021-02-11T00:24:35.888 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-11T00:24:35.890 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.925 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.041) 0:01:47.351 ***** 2021-02-11T00:24:35.932 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T00:24:35.933 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:35.974 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:35 +0000 (0:00:00.048) 0:01:47.400 ***** 2021-02-11T00:24:35.980 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-11T00:24:35.981 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:36.017 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.043) 0:01:47.444 ***** 2021-02-11T00:24:36.024 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T00:24:36.026 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:36.062 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.044) 0:01:47.488 ***** 2021-02-11T00:24:36.069 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-11T00:24:36.070 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:36.105 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.043) 0:01:47.532 ***** 2021-02-11T00:24:36.112 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T00:24:36.113 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:36.151 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.045) 0:01:47.577 ***** 2021-02-11T00:24:36.157 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-11T00:24:36.158 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:36.205 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.053) 0:01:47.631 ***** 2021-02-11T00:24:36.213 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T00:24:36.215 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:36.256 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.051) 0:01:47.683 ***** 2021-02-11T00:24:36.265 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T00:24:36.266 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:36.304 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.048) 0:01:47.731 ***** 2021-02-11T00:24:36.311 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T00:24:36.312 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:36.353 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.048) 0:01:47.780 ***** 2021-02-11T00:24:36.363 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-11T00:24:36.365 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:36.400 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.046) 0:01:47.826 ***** 2021-02-11T00:24:36.405 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-02-11T00:24:36.407 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:36.447 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.047) 0:01:47.873 ***** 2021-02-11T00:24:36.452 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-11T00:24:36.454 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:36.492 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.045) 0:01:47.918 ***** 2021-02-11T00:24:36.521 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-11T00:24:36.522 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:36.555 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.063) 0:01:47.982 ***** 2021-02-11T00:24:36.572 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-02-11T00:24:36.573 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-02-11T00:24:36.614 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.058) 0:01:48.041 ***** 2021-02-11T00:24:36.624 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-11T00:24:36.624 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:36.686 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.072) 0:01:48.113 ***** 2021-02-11T00:24:36.713 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T00:24:36.714 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:36.749 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.062) 0:01:48.175 ***** 2021-02-11T00:24:36.775 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T00:24:36.777 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:36.811 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.062) 0:01:48.237 ***** 2021-02-11T00:24:36.837 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T00:24:36.838 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:36.873 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.061) 0:01:48.299 ***** 2021-02-11T00:24:36.900 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-11T00:24:36.902 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:36.936 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.063) 0:01:48.363 ***** 2021-02-11T00:24:36.942 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-11T00:24:36.943 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-02-11T00:24:36.978 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:36 +0000 (0:00:00.042) 0:01:48.405 ***** 2021-02-11T00:24:36.984 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-11T00:24:36.986 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:37.022 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:37 +0000 (0:00:00.043) 0:01:48.448 ***** 2021-02-11T00:24:37.028 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-11T00:24:37.030 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:37.064 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:37 +0000 (0:00:00.042) 0:01:48.491 ***** 2021-02-11T00:24:37.071 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-11T00:24:37.072 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:37.108 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:37 +0000 (0:00:00.043) 0:01:48.534 ***** 2021-02-11T00:24:37.134 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T00:24:37.136 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:37.171 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:37 +0000 (0:00:00.063) 0:01:48.597 ***** 2021-02-11T00:24:37.197 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T00:24:37.198 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:37.237 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:37 +0000 (0:00:00.066) 0:01:48.664 ***** 2021-02-11T00:24:37.264 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:37.300 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:37 +0000 (0:00:00.062) 0:01:48.726 ***** 2021-02-11T00:24:37.326 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-11T00:24:37.327 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:37.361 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:37 +0000 (0:00:00.061) 0:01:48.788 ***** 2021-02-11T00:24:37.387 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-11T00:24:37.388 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Upload weekly kernel-clean crontab.] ************************** ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-02-11T00:24:37.423 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:37 +0000 (0:00:00.062) 0:01:48.850 ***** 2021-02-11T00:24:37.456 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:37 +0000 (0:00:00.033) 0:01:48.883 ***** 2021-02-11T00:24:37.464 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-11T00:24:37.464 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:37.500 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:37 +0000 (0:00:00.043) 0:01:48.927 ***** 2021-02-11T00:24:37.527 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-02-11T00:24:37.528 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:37.562 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:37 +0000 (0:00:00.061) 0:01:48.989 ***** 2021-02-11T00:24:37.713 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-02-11T00:24:37.715 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-02-11T00:24:37.752 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:37 +0000 (0:00:00.189) 0:01:49.179 ***** 2021-02-11T00:24:37.780 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-02-11T00:24:37.780 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-02-11T00:24:37.817 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:37 +0000 (0:00:00.065) 0:01:49.244 ***** 2021-02-11T00:24:38.053 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-11T00:24:38.053 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-02-11T00:24:38.094 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:38 +0000 (0:00:00.276) 0:01:49.521 ***** 2021-02-11T00:24:38.123 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-02-11T00:24:38.124 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.160 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:38 +0000 (0:00:00.066) 0:01:49.587 ***** 2021-02-11T00:24:38.194 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:38 +0000 (0:00:00.034) 0:01:49.621 ***** 2021-02-11T00:24:38.335 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-02-11T00:24:38.337 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T00:24:38.377 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:38 +0000 (0:00:00.183) 0:01:49.804 ***** 2021-02-11T00:24:38.404 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-02-11T00:24:38.405 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.445 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:38 +0000 (0:00:00.067) 0:01:49.871 ***** 2021-02-11T00:24:38.473 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-02-11T00:24:38.475 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-02-11T00:24:38.508 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:38 +0000 (0:00:00.063) 0:01:49.935 ***** 2021-02-11T00:24:38.515 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-02-11T00:24:38.516 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.551 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:38 +0000 (0:00:00.042) 0:01:49.978 ***** 2021-02-11T00:24:38.577 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-02-11T00:24:38.578 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.611 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:38 +0000 (0:00:00.059) 0:01:50.038 ***** 2021-02-11T00:24:38.638 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-02-11T00:24:38.639 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.673 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:38 +0000 (0:00:00.061) 0:01:50.099 ***** 2021-02-11T00:24:38.700 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-02-11T00:24:38.701 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.735 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:38 +0000 (0:00:00.061) 0:01:50.161 ***** 2021-02-11T00:24:38.762 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T00:24:38.763 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-02-11T00:24:38.796 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:38 +0000 (0:00:00.061) 0:01:50.223 ***** 2021-02-11T00:24:38.812 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-02-11T00:24:38.812 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-02-11T00:24:38.849 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:38 +0000 (0:00:00.052) 0:01:50.275 ***** 2021-02-11T00:24:38.856 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-02-11T00:24:38.857 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.858 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi095.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.859 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.870 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.881 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.892 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-02-11T00:24:38.892 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.903 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.914 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.914 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.925 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.936 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.937 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.947 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:38.988 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:38 +0000 (0:00:00.139) 0:01:50.415 ***** 2021-02-11T00:24:39.170 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-02-11T00:24:39.171 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-02-11T00:24:39.208 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:39 +0000 (0:00:00.219) 0:01:50.634 ***** 2021-02-11T00:24:39.214 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-02-11T00:24:39.214 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004433", "end": "2021-02-11 00:24:39.168075", "rc": 0, "start": "2021-02-11 00:24:39.163642", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock"]} 2021-02-11T00:24:39.255 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:39 +0000 (0:00:00.047) 0:01:50.681 ***** 2021-02-11T00:24:39.291 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-02-11T00:24:39.292 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.336 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:39 +0000 (0:00:00.081) 0:01:50.763 ***** 2021-02-11T00:24:39.342 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-02-11T00:24:39.343 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T00:24:39.380 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:39 +0000 (0:00:00.043) 0:01:50.807 ***** 2021-02-11T00:24:39.387 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-02-11T00:24:39.388 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.427 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:39 +0000 (0:00:00.046) 0:01:50.853 ***** 2021-02-11T00:24:39.432 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-02-11T00:24:39.433 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.469 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:39 +0000 (0:00:00.042) 0:01:50.896 ***** 2021-02-11T00:24:39.475 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-02-11T00:24:39.477 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.511 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:39 +0000 (0:00:00.041) 0:01:50.938 ***** 2021-02-11T00:24:39.517 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-02-11T00:24:39.517 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.554 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:39 +0000 (0:00:00.042) 0:01:50.981 ***** 2021-02-11T00:24:39.580 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-02-11T00:24:39.581 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.616 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:39 +0000 (0:00:00.061) 0:01:51.042 ***** 2021-02-11T00:24:39.642 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-02-11T00:24:39.643 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.677 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:39 +0000 (0:00:00.061) 0:01:51.104 ***** 2021-02-11T00:24:39.704 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-02-11T00:24:39.705 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.739 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:39 +0000 (0:00:00.061) 0:01:51.166 ***** 2021-02-11T00:24:39.766 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T00:24:39.767 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-02-11T00:24:39.802 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:39 +0000 (0:00:00.062) 0:01:51.229 ***** 2021-02-11T00:24:39.849 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-02-11T00:24:39.851 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.860 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.861 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.871 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.883 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi057.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.904 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-02-11T00:24:39.904 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.915 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.915 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.926 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.927 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.938 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.949 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} skipping: [smithi057.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-11T00:24:39.993 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:39 +0000 (0:00:00.190) 0:01:51.420 ***** 2021-02-11T00:24:40.194 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-02-11T00:24:40.195 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003171", "end": "2021-02-11 00:24:40.170189", "rc": 0, "start": "2021-02-11 00:24:40.167018", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock"]} 2021-02-11T00:24:40.241 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:40 +0000 (0:00:00.247) 0:01:51.667 ***** 2021-02-11T00:24:40.270 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-02-11T00:24:40.272 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:40.311 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:40 +0000 (0:00:00.070) 0:01:51.737 ***** 2021-02-11T00:24:40.338 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-02-11T00:24:40.339 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:40.376 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:40 +0000 (0:00:00.065) 0:01:51.803 ***** 2021-02-11T00:24:40.392 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-02-11T00:24:40.394 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"cache_update_time": 1613002785, "cache_updated": false, "changed": false} 2021-02-11T00:24:40.431 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:40 +0000 (0:00:00.055) 0:01:51.858 ***** 2021-02-11T00:24:40.437 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-02-11T00:24:40.439 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:40.476 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:40 +0000 (0:00:00.044) 0:01:51.903 ***** 2021-02-11T00:24:40.483 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-02-11T00:24:40.485 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:40.525 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:40 +0000 (0:00:00.048) 0:01:51.951 ***** 2021-02-11T00:24:40.561 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-11T00:24:40.562 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item={'block_used': 13754869, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340239, 'size_available': 927089618944, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340239, "block_size": 4096, "block_total": 240095108, "block_used": 13754869, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089618944, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}} 2021-02-11T00:24:40.597 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:40 +0000 (0:00:00.072) 0:01:52.024 ***** 2021-02-11T00:24:40.729 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-02-11T00:24:40.730 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.011029", "end": "2021-02-11 00:24:40.737003", "rc": 0, "start": "2021-02-11 00:24:40.725974", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:40.766 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:40 +0000 (0:00:00.168) 0:01:52.192 ***** 2021-02-11T00:24:40.803 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-02-11T00:24:40.805 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'block_used': 13754869, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340239, 'size_available': 927089618944, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340239, "block_size": 4096, "block_total": 240095108, "block_used": 13754869, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089618944, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:40.842 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:40 +0000 (0:00:00.075) 0:01:52.268 ***** 2021-02-11T00:24:40.868 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-02-11T00:24:40.904 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:40 +0000 (0:00:00.062) 0:01:52.331 ***** 2021-02-11T00:24:40.953 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-11T00:24:40.954 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:41.393 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-02-11T00:24:41.394 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"cache_update_time": 1613002741, "cache_updated": false, "changed": false} 2021-02-11T00:24:41.432 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:41 +0000 (0:00:00.527) 0:01:52.858 ***** 2021-02-11T00:24:41.464 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-02-11T00:24:41.466 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:41.501 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:41 +0000 (0:00:00.068) 0:01:52.927 ***** 2021-02-11T00:24:41.550 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-11T00:24:41.552 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item={'block_used': 13756920, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226338188, 'size_available': 927081218048, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226338188, "block_size": 4096, "block_total": 240095108, "block_used": 13756920, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927081218048, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}} 2021-02-11T00:24:41.587 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:41 +0000 (0:00:00.086) 0:01:53.014 ***** 2021-02-11T00:24:41.741 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-02-11T00:24:41.742 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009849", "end": "2021-02-11 00:24:41.899610", "rc": 0, "start": "2021-02-11 00:24:41.889761", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:41.797 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:41 +0000 (0:00:00.209) 0:01:53.223 ***** 2021-02-11T00:24:41.839 INFO:teuthology.task.ansible.out: 2021-02-11T00:24:41.841 INFO:teuthology.task.ansible.out:TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi057.front.sepia.ceph.com] => (item={'block_used': 13756920, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226338188, 'size_available': 927081218048, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226338188, "block_size": 4096, "block_total": 240095108, "block_used": 13756920, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927081218048, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:41.878 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:41 +0000 (0:00:00.081) 0:01:53.305 ***** 2021-02-11T00:24:41.905 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-02-11T00:24:41.942 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:41 +0000 (0:00:00.063) 0:01:53.368 ***** 2021-02-11T00:24:42.003 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-11T00:24:42.003 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:43.071 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WNBLTMxNDKRAbRBcMe0VkE1KFkh6CDoT0'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:02.004720", "end": "2021-02-11 00:24:43.078384", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WNBLTMxNDKRAbRBcMe0VkE1KFkh6CDoT0"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:41.073664", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T00:24:43.083 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:43.094 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:44.064 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5THAAXphMOj7yFNgdxankACPqqdb1f5Kat'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.941562", "end": "2021-02-11 00:24:44.222221", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5THAAXphMOj7yFNgdxankACPqqdb1f5Kat"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:42.280659", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T00:24:44.076 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:44.091 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:44.339 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WKaODJCMS2m15e4uuQEmLkl8KQ1pEXJFF'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.123561", "end": "2021-02-11 00:24:44.336581", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WKaODJCMS2m15e4uuQEmLkl8KQ1pEXJFF"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:43.213020", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T00:24:44.351 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJRM', 'wwn-0x5000c500918dc09f'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJRM-part1', 'wwn-0x5000c500918dc09f-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918dc09f', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LJRM', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJRM", "wwn-0x5000c500918dc09f"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJRM-part1", "wwn-0x5000c500918dc09f-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LJRM", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918dc09f"}}, "skip_reason": 2021-02-11T00:24:44.352 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-11T00:24:45.336 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5TsxoBWh77lM39LJEUFmYEpjPUTxz0fYCp'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.114779", "end": "2021-02-11 00:24:45.487124", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5TsxoBWh77lM39LJEUFmYEpjPUTxz0fYCp"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:44.372345", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T00:24:45.348 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQFM', 'wwn-0x5000c500871a80ed'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQFM-part1', 'wwn-0x5000c500871a80ed-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871a80ed', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HQFM', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQFM", "wwn-0x5000c500871a80ed"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQFM-part1", "wwn-0x5000c500871a80ed-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HQFM", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a80ed"}}, "skip_reason": 2021-02-11T00:24:45.349 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-11T00:24:45.619 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-S6Zkou-37jZ-rttT-0Rbm-NGzu-7pIY-sm0Epm', 'nvme-INTEL_SSDPEDMD400G4_PHFT6204010G400BGN', 'nvme-nvme.8086-50484654363230343031304734303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.148926", "end": "2021-02-11 00:24:45.612934", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-S6Zkou-37jZ-rttT-0Rbm-NGzu-7pIY-sm0Epm", "nvme-INTEL_SSDPEDMD400G4_PHFT6204010G400BGN", "nvme-nvme.8086-50484654363230343031304734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:44.464008", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-02-11T00:24:45.620 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T00:24:45.630 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:45.631 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:46.754 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '7814037168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-5DOeyk-k3u1-sgDR-vZi5-0OsY-hrUf-ziiOax', 'nvme-INTEL_SSDPEDKE040T7_BTLE7361049H4P0KGN', 'nvme-eui.0100000001000000e4d25c000034df14'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Express Flash NVMe P4500/P4600', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDKE040T7', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '3.64 TB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.283454", "end": "2021-02-11 00:24:46.905172", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Express Flash NVMe P4500/P4600", "links": {"ids": ["lvm-pv-uuid-5DOeyk-k3u1-sgDR-vZi5-0OsY-hrUf-ziiOax", "nvme-INTEL_SSDPEDKE040T7_BTLE7361049H4P0KGN", "nvme-eui.0100000001000000e4d25c000034df14"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDKE040T7", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "7814037168", "sectorsize": "512", "size": "3.64 TB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:45.621718", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT 2021-02-11T00:24:46.756 INFO:teuthology.task.ansible.out: data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T00:24:46.766 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:46.777 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:46.914 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WBYxe2ZcLc4xQBelXvur0faXgQgpZC3qI'], 'uuids': ['41a77112-e02a-4307-86e8-3612dc6e33a0']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.166650", "end": "2021-02-11 00:24:46.912181", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WBYxe2ZcLc4xQBelXvur0faXgQgpZC3qI"], "labels": [], "masters": [], "uuids": ["41a77112-e02a-4307-86e8-3612dc6e33a0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:45.745531", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T00:24:46.926 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:46.937 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:48.034 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5Tvdf9h5jSCtTK0xbqdQ6GIOgk1NaoR8Cq'], 'uuids': ['a4ff6139-1279-4c19-96d6-5e3e5587d5f7']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.132526", "end": "2021-02-11 00:24:48.178526", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5Tvdf9h5jSCtTK0xbqdQ6GIOgk1NaoR8Cq"], "labels": [], "masters": [], "uuids": ["a4ff6139-1279-4c19-96d6-5e3e5587d5f7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:47.046000", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T00:24:48.035 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:48.047 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:48.203 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WWmA0n3UVTj52QzzGneVhLjVirz1AWsX6'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.152308", "end": "2021-02-11 00:24:48.204442", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WWmA0n3UVTj52QzzGneVhLjVirz1AWsX6"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:47.052134", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T00:24:48.216 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:49.279 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5Tdb8frCOBTa2kfuwNdxt23y5PcynzPdei'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.110335", "end": "2021-02-11 00:24:49.436197", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5Tdb8frCOBTa2kfuwNdxt23y5PcynzPdei"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:48.325862", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T00:24:49.291 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:49.480 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WxjNczR8759WdB99Fi1vJsoneGWODSM40'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.147391", "end": "2021-02-11 00:24:49.480148", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WxjNczR8759WdB99Fi1vJsoneGWODSM40"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:48.332757", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T00:24:49.533 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:49 +0000 (0:00:07.591) 0:02:00.959 ***** 2021-02-11T00:24:49.582 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T00:24:49.583 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:49.854 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WNBLTMxNDKRAbRBcMe0VkE1KFkh6CDoT0'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.153344", "end": "2021-02-11 00:24:49.854085", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WNBLTMxNDKRAbRBcMe0VkE1KFkh6CDoT0"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:49.700741", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148725 s, 776 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148725 s, 776 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:49.866 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:49.877 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:50.160 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WKaODJCMS2m15e4uuQEmLkl8KQ1pEXJFF'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.159935", "end": "2021-02-11 00:24:50.160156", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WKaODJCMS2m15e4uuQEmLkl8KQ1pEXJFF"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:50.000221", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.155031 s, 744 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.155031 s, 744 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:50.172 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJRM', 'wwn-0x5000c500918dc09f'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJRM-part1', 'wwn-0x5000c500918dc09f-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918dc09f', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LJRM', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJRM", "wwn-0x5000c500918dc09f"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJRM-part1", "wwn-0x5000c500918dc09f-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LJRM", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918dc09f"}}, "skip_reason": 2021-02-11T00:24:50.173 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-11T00:24:50.350 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-S6Zkou-37jZ-rttT-0Rbm-NGzu-7pIY-sm0Epm', 'nvme-INTEL_SSDPEDMD400G4_PHFT6204010G400BGN', 'nvme-nvme.8086-50484654363230343031304734303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.051380", "end": "2021-02-11 00:24:50.346938", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-S6Zkou-37jZ-rttT-0Rbm-NGzu-7pIY-sm0Epm", "nvme-INTEL_SSDPEDMD400G4_PHFT6204010G400BGN", "nvme-nvme.8086-50484654363230343031304734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:50.295558", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0478161 s, 2.4 GB/s", "std 2021-02-11T00:24:50.351 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0478161 s, 2.4 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:50.362 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:50.373 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:50.529 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-11T00:24:50.530 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5TJ7ZfvQ640eFLwXadpDkWb1N1fxqE60gC'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.118474", "end": "2021-02-11 00:24:50.686954", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5TJ7ZfvQ640eFLwXadpDkWb1N1fxqE60gC"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:49.568480", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-11T00:24:50.575 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:50 +0000 (0:00:01.042) 0:02:02.002 ***** 2021-02-11T00:24:50.623 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T00:24:50.625 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:50.645 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WBYxe2ZcLc4xQBelXvur0faXgQgpZC3qI'], 'uuids': ['41a77112-e02a-4307-86e8-3612dc6e33a0']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.148405", "end": "2021-02-11 00:24:50.636710", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WBYxe2ZcLc4xQBelXvur0faXgQgpZC3qI"], "labels": [], "masters": [], "uuids": ["41a77112-e02a-4307-86e8-3612dc6e33a0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:50.488305", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144169 s, 800 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144169 s, 800 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:50.657 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:50.668 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:50.908 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5THAAXphMOj7yFNgdxankACPqqdb1f5Kat'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.160434", "end": "2021-02-11 00:24:51.061143", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5THAAXphMOj7yFNgdxankACPqqdb1f5Kat"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:50.900709", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.156321 s, 738 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.156321 s, 738 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:50.919 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:50.932 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:50.965 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WWmA0n3UVTj52QzzGneVhLjVirz1AWsX6'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.159099", "end": "2021-02-11 00:24:50.955739", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WWmA0n3UVTj52QzzGneVhLjVirz1AWsX6"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:50.796640", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.15432 s, 747 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.15432 s, 747 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:50.976 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:51.210 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5TsxoBWh77lM39LJEUFmYEpjPUTxz0fYCp'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.163813", "end": "2021-02-11 00:24:51.358434", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5TsxoBWh77lM39LJEUFmYEpjPUTxz0fYCp"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:51.194621", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.159187 s, 725 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.159187 s, 725 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:51.212 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQFM', 'wwn-0x5000c500871a80ed'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQFM-part1', 'wwn-0x5000c500871a80ed-part1'], 'uuids': ['602a9f0b-ccf6-4d5d-877a-542e0952c07a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871a80ed', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HQFM', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQFM", "wwn-0x5000c500871a80ed"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQFM-part1", "wwn-0x5000c500871a80ed-part1"], "labels": [], "masters": [], "uuids": ["602a9f0b-ccf6-4d5d-877a-542e0952c07a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HQFM", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a80ed"}}, "skip_reason": 2021-02-11T00:24:51.213 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-02-11T00:24:51.266 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WxjNczR8759WdB99Fi1vJsoneGWODSM40'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.158241", "end": "2021-02-11 00:24:51.266253", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-lAt5AizZlxIV73kyxsv7fclma5DSAc0WxjNczR8759WdB99Fi1vJsoneGWODSM40"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:51.108012", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153786 s, 750 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153786 s, 750 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:51.313 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:51 +0000 (0:00:00.736) 0:02:02.739 ***** 2021-02-11T00:24:51.384 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '7814037168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-5DOeyk-k3u1-sgDR-vZi5-0OsY-hrUf-ziiOax', 'nvme-INTEL_SSDPEDKE040T7_BTLE7361049H4P0KGN', 'nvme-eui.0100000001000000e4d25c000034df14'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Express Flash NVMe P4500/P4600', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDKE040T7', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '3.64 TB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.047059", "end": "2021-02-11 00:24:51.534804", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Express Flash NVMe P4500/P4600", "links": {"ids": ["lvm-pv-uuid-5DOeyk-k3u1-sgDR-vZi5-0OsY-hrUf-ziiOax", "nvme-INTEL_SSDPEDKE040T7_BTLE7361049H4P0KGN", "nvme-eui.0100000001000000e4d25c000034df14"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDKE040T7", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "7814037168", "sectorsize": "512", "size": "3.64 TB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:51.487745", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0435725 s, 2.6 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0435725 s, 2.6 GB 2021-02-11T00:24:51.386 INFO:teuthology.task.ansible.out:/s"], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:51.387 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:51.398 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:51.599 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-02-11T00:24:51.600 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.144874", "end": "2021-02-11 00:24:51.592850", "rc": 0, "start": "2021-02-11 00:24:51.447976", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:51.636 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:51 +0000 (0:00:00.323) 0:02:03.062 ***** 2021-02-11T00:24:51.716 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T00:24:51.717 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5Tvdf9h5jSCtTK0xbqdQ6GIOgk1NaoR8Cq'], 'uuids': ['a4ff6139-1279-4c19-96d6-5e3e5587d5f7']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.192594", "end": "2021-02-11 00:24:51.860945", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5Tvdf9h5jSCtTK0xbqdQ6GIOgk1NaoR8Cq"], "labels": [], "masters": [], "uuids": ["a4ff6139-1279-4c19-96d6-5e3e5587d5f7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:51.668351", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.189673 s, 608 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.189673 s, 608 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:51.728 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:51.729 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:51.835 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-02-11T00:24:51.837 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.060874", "end": "2021-02-11 00:24:51.836729", "rc": 0, "start": "2021-02-11 00:24:51.775855", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:51.872 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:51 +0000 (0:00:00.235) 0:02:03.298 ***** 2021-02-11T00:24:51.900 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-02-11T00:24:51.937 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:51 +0000 (0:00:00.065) 0:02:03.363 ***** 2021-02-11T00:24:51.963 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:52.001 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:51 +0000 (0:00:00.063) 0:02:03.427 ***** 2021-02-11T00:24:52.008 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T00:24:52.008 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5Tdb8frCOBTa2kfuwNdxt23y5PcynzPdei'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.150982", "end": "2021-02-11 00:24:52.152349", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5Tdb8frCOBTa2kfuwNdxt23y5PcynzPdei"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:52.001367", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.146984 s, 785 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.146984 s, 785 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:52.010 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:52.032 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:52.072 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:52 +0000 (0:00:00.071) 0:02:03.498 ***** 2021-02-11T00:24:52.100 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-11T00:24:52.102 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:52.138 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:52 +0000 (0:00:00.066) 0:02:03.565 ***** 2021-02-11T00:24:52.165 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-02-11T00:24:52.166 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:52.200 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:52 +0000 (0:00:00.061) 0:02:03.627 ***** 2021-02-11T00:24:52.306 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-11T00:24:52.306 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5TJ7ZfvQ640eFLwXadpDkWb1N1fxqE60gC'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.166203", "end": "2021-02-11 00:24:52.450227", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-42wehx8CcooI0RcUHEW72fxnX0Rn6c5TJ7ZfvQ640eFLwXadpDkWb1N1fxqE60gC"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-11 00:24:52.284024", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.161541 s, 714 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.161541 s, 714 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:52.352 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:52 +0000 (0:00:00.151) 0:02:03.778 ***** 2021-02-11T00:24:52.359 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-02-11T00:24:52.360 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T00:24:52.398 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:52 +0000 (0:00:00.046) 0:02:03.824 ***** 2021-02-11T00:24:52.435 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-11T00:24:52.435 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'block_used': 13754869, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226340239, 'size_available': 927089618944, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226340239, "block_size": 4096, "block_total": 240095108, "block_used": 13754869, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927089618944, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:52.471 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:52 +0000 (0:00:00.072) 0:02:03.897 ***** 2021-02-11T00:24:52.497 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-02-11T00:24:52.499 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:52.531 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:52 +0000 (0:00:00.060) 0:02:03.958 ***** 2021-02-11T00:24:52.558 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T00:24:52.559 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:52.592 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:52 +0000 (0:00:00.060) 0:02:04.019 ***** 2021-02-11T00:24:52.618 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T00:24:52.619 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:52.654 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:52 +0000 (0:00:00.062) 0:02:04.081 ***** 2021-02-11T00:24:52.662 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-02-11T00:24:52.662 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.127194", "end": "2021-02-11 00:24:52.773103", "rc": 0, "start": "2021-02-11 00:24:52.645909", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:52.699 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:52 +0000 (0:00:00.044) 0:02:04.125 ***** 2021-02-11T00:24:52.704 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-02-11T00:24:52.706 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:52.742 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:52 +0000 (0:00:00.043) 0:02:04.169 ***** 2021-02-11T00:24:52.769 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-02-11T00:24:52.770 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:52.805 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:52 +0000 (0:00:00.062) 0:02:04.231 ***** 2021-02-11T00:24:52.831 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-02-11T00:24:52.832 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:52.866 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:52 +0000 (0:00:00.061) 0:02:04.293 ***** 2021-02-11T00:24:52.873 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-02-11T00:24:52.873 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.046390", "end": "2021-02-11 00:24:53.027844", "rc": 0, "start": "2021-02-11 00:24:52.981454", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:24:52.910 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:52 +0000 (0:00:00.043) 0:02:04.337 ***** 2021-02-11T00:24:52.937 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-02-11T00:24:52.980 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:52 +0000 (0:00:00.069) 0:02:04.406 ***** 2021-02-11T00:24:53.007 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-02-11T00:24:53.008 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:53.045 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:53 +0000 (0:00:00.065) 0:02:04.472 ***** 2021-02-11T00:24:53.073 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-02-11T00:24:53.075 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:53.110 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:53 +0000 (0:00:00.064) 0:02:04.536 ***** 2021-02-11T00:24:53.136 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-11T00:24:53.138 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:53.175 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:53 +0000 (0:00:00.065) 0:02:04.601 ***** 2021-02-11T00:24:53.202 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-02-11T00:24:53.203 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:53.237 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:53 +0000 (0:00:00.062) 0:02:04.664 ***** 2021-02-11T00:24:53.359 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-02-11T00:24:53.361 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-11T00:24:53.397 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:53 +0000 (0:00:00.159) 0:02:04.823 ***** 2021-02-11T00:24:53.435 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-02-11T00:24:53.436 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} TASK [testnode : Set root disk] ************************************************ 2021-02-11T00:24:53.437 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'block_used': 13756920, 'uuid': '602a9f0b-ccf6-4d5d-877a-542e0952c07a', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226338188, 'size_available': 927081218048, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149545, 'block_size': 4096, 'inode_available': 60905431}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226338188, "block_size": 4096, "block_total": 240095108, "block_used": 13756920, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905431, "inode_total": 61054976, "inode_used": 149545, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927081218048, "size_total": 983429562368, "uuid": "602a9f0b-ccf6-4d5d-877a-542e0952c07a"}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:53.478 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:53 +0000 (0:00:00.081) 0:02:04.905 ***** 2021-02-11T00:24:53.514 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:53 +0000 (0:00:00.036) 0:02:04.941 ***** 2021-02-11T00:24:53.522 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-02-11T00:24:53.523 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:53.559 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:53 +0000 (0:00:00.044) 0:02:04.986 ***** 2021-02-11T00:24:53.587 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T00:24:53.589 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:53.624 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:53 +0000 (0:00:00.065) 0:02:05.051 ***** 2021-02-11T00:24:53.653 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T00:24:53.655 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:53.690 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:53 +0000 (0:00:00.065) 0:02:05.116 ***** 2021-02-11T00:24:53.719 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-02-11T00:24:53.720 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:53.756 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:53 +0000 (0:00:00.066) 0:02:05.183 ***** 2021-02-11T00:24:53.794 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-02-11T00:24:53.795 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:53.839 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:53 +0000 (0:00:00.082) 0:02:05.265 ***** 2021-02-11T00:24:53.865 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-02-11T00:24:53.866 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:53.902 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:53 +0000 (0:00:00.063) 0:02:05.329 ***** 2021-02-11T00:24:54.108 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T00:24:54.109 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T00:24:54.203 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-02-11T00:24:54.204 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-11T00:24:54.239 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:54 +0000 (0:00:00.336) 0:02:05.666 ***** 2021-02-11T00:24:54.339 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T00:24:54.340 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T00:24:54.517 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T00:24:54.581 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T00:24:54.748 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T00:24:54.822 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T00:24:54.990 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T00:24:55.064 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T00:24:55.102 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:55 +0000 (0:00:00.863) 0:02:06.529 ***** 2021-02-11T00:24:55.193 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T00:24:55.236 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-11T00:24:55.237 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-11T00:24:55.272 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:55 +0000 (0:00:00.169) 0:02:06.699 ***** 2021-02-11T00:24:55.424 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T00:24:55.426 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T00:24:55.436 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-11T00:24:55.437 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-11T00:24:55.475 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:55 +0000 (0:00:00.202) 0:02:06.901 ***** 2021-02-11T00:24:55.534 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T00:24:55.535 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T00:24:55.589 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-11T00:24:55.591 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-11T00:24:55.624 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:55 +0000 (0:00:00.149) 0:02:07.051 ***** 2021-02-11T00:24:55.641 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T00:24:55.641 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T00:24:55.757 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T00:24:55.768 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:55.773 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T00:24:55.810 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:55 +0000 (0:00:00.185) 0:02:07.237 ***** 2021-02-11T00:24:55.837 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-02-11T00:24:55.838 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:55.875 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:55 +0000 (0:00:00.065) 0:02:07.302 ***** 2021-02-11T00:24:55.892 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T00:24:55.892 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T00:24:55.904 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-02-11T00:24:55.905 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:55.940 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:55 +0000 (0:00:00.065) 0:02:07.367 ***** 2021-02-11T00:24:55.998 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T00:24:56.000 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T00:24:56.073 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-02-11T00:24:56.074 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T00:24:56.116 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:56 +0000 (0:00:00.175) 0:02:07.542 ***** 2021-02-11T00:24:56.132 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-11T00:24:56.133 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-11T00:24:56.134 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:56.172 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:56 +0000 (0:00:00.056) 0:02:07.598 ***** 2021-02-11T00:24:56.178 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-02-11T00:24:56.178 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-02-11T00:24:56.217 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:56 +0000 (0:00:00.044) 0:02:07.643 ***** 2021-02-11T00:24:56.223 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-02-11T00:24:56.223 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:56.260 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:56 +0000 (0:00:00.043) 0:02:07.687 ***** 2021-02-11T00:24:56.287 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-02-11T00:24:56.289 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:56.323 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:56 +0000 (0:00:00.062) 0:02:07.749 ***** 2021-02-11T00:24:56.464 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-02-11T00:24:56.465 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T00:24:56.502 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:56 +0000 (0:00:00.179) 0:02:07.928 ***** 2021-02-11T00:24:56.539 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-02-11T00:24:56.540 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-02-11T00:24:56.575 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:56 +0000 (0:00:00.073) 0:02:08.002 ***** 2021-02-11T00:24:56.697 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-02-11T00:24:56.698 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"changed": true} 2021-02-11T00:24:56.736 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:56 +0000 (0:00:00.161) 0:02:08.163 ***** 2021-02-11T00:24:57.213 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-02-11T00:24:57.214 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-02-11T00:24:57.254 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:57 +0000 (0:00:00.517) 0:02:08.681 ***** 2021-02-11T00:24:57.280 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-02-11T00:24:57.282 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:57.316 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:57 +0000 (0:00:00.061) 0:02:08.742 ***** 2021-02-11T00:24:57.341 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-02-11T00:24:57.342 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:57.377 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:57 +0000 (0:00:00.060) 0:02:08.803 ***** 2021-02-11T00:24:57.404 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-11T00:24:57.405 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:57.440 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:57 +0000 (0:00:00.062) 0:02:08.866 ***** 2021-02-11T00:24:57.466 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-11T00:24:57.467 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:57.503 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:57 +0000 (0:00:00.062) 0:02:08.929 ***** 2021-02-11T00:24:57.643 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-02-11T00:24:57.645 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-02-11T00:24:57.679 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:57 +0000 (0:00:00.177) 0:02:09.106 ***** 2021-02-11T00:24:57.809 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-02-11T00:24:57.810 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T00:24:57.845 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:57 +0000 (0:00:00.165) 0:02:09.271 ***** 2021-02-11T00:24:57.975 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-02-11T00:24:57.976 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true} 2021-02-11T00:24:58.012 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:58 +0000 (0:00:00.167) 0:02:09.439 ***** 2021-02-11T00:24:58.154 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-02-11T00:24:58.155 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-02-11T00:24:58.192 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:58 +0000 (0:00:00.180) 0:02:09.619 ***** 2021-02-11T00:24:58.209 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-02-11T00:24:58.211 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-02-11T00:24:58.247 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:58 +0000 (0:00:00.054) 0:02:09.674 ***** 2021-02-11T00:24:58.273 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-02-11T00:24:58.275 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:58.309 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:58 +0000 (0:00:00.061) 0:02:09.736 ***** 2021-02-11T00:24:58.335 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-02-11T00:24:58.337 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:58.370 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:58 +0000 (0:00:00.060) 0:02:09.797 ***** 2021-02-11T00:24:58.407 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:58.447 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:58 +0000 (0:00:00.077) 0:02:09.874 ***** 2021-02-11T00:24:58.478 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-11T00:24:58.480 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:58.516 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:58 +0000 (0:00:00.068) 0:02:09.943 ***** 2021-02-11T00:24:58.523 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-02-11T00:24:58.524 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-02-11T00:24:58.561 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:58 +0000 (0:00:00.044) 0:02:09.988 ***** 2021-02-11T00:24:58.672 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-02-11T00:24:58.674 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-02-11T00:24:58.710 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:58 +0000 (0:00:00.149) 0:02:10.137 ***** 2021-02-11T00:24:58.841 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-02-11T00:24:58.843 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T00:24:58.881 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:58 +0000 (0:00:00.170) 0:02:10.307 ***** 2021-02-11T00:24:59.201 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-02-11T00:24:59.203 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-02-11T00:24:59.238 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:59 +0000 (0:00:00.357) 0:02:10.665 ***** 2021-02-11T00:24:59.494 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-02-11T00:24:59.495 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"cache_update_time": 1613002785, "cache_updated": false, "changed": false} 2021-02-11T00:24:59.543 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:59 +0000 (0:00:00.304) 0:02:10.970 ***** 2021-02-11T00:24:59.559 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-02-11T00:24:59.560 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-02-11T00:24:59.597 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:59 +0000 (0:00:00.053) 0:02:11.024 ***** 2021-02-11T00:24:59.718 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-02-11T00:24:59.719 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.047074", "end": "2021-02-11 00:24:59.728114", "rc": 0, "start": "2021-02-11 00:24:59.681040", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-02-11T00:24:59.756 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:59 +0000 (0:00:00.159) 0:02:11.183 ***** 2021-02-11T00:24:59.784 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-02-11T00:24:59.786 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:59.821 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:59 +0000 (0:00:00.064) 0:02:11.247 ***** 2021-02-11T00:24:59.848 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:24:59.891 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:24:59 +0000 (0:00:00.069) 0:02:11.317 ***** 2021-02-11T00:25:00.567 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-02-11T00:25:00.568 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"cache_update_time": 1613002741, "cache_updated": false, "changed": false} 2021-02-11T00:25:00.619 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:00 +0000 (0:00:00.728) 0:02:12.046 ***** 2021-02-11T00:25:00.773 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-02-11T00:25:00.774 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"cache_update_time": 1613002785, "cache_updated": false, "changed": false} 2021-02-11T00:25:00.811 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:00 +0000 (0:00:00.192) 0:02:12.238 ***** 2021-02-11T00:25:00.817 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-02-11T00:25:00.817 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.045842", "end": "2021-02-11 00:25:00.964814", "rc": 0, "start": "2021-02-11 00:25:00.918972", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-02-11T00:25:00.855 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:00 +0000 (0:00:00.044) 0:02:12.282 ***** 2021-02-11T00:25:00.866 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-02-11T00:25:00.867 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:00.920 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:00 +0000 (0:00:00.064) 0:02:12.347 ***** 2021-02-11T00:25:00.928 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-02-11T00:25:00.929 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:00.992 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:00 +0000 (0:00:00.071) 0:02:12.418 ***** 2021-02-11T00:25:01.024 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-02-11T00:25:01.026 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:01.074 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:01 +0000 (0:00:00.081) 0:02:12.500 ***** 2021-02-11T00:25:01.460 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-02-11T00:25:01.462 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-02-11T00:25:01.501 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:01 +0000 (0:00:00.427) 0:02:12.928 ***** 2021-02-11T00:25:01.528 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-02-11T00:25:01.529 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:01.564 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:01 +0000 (0:00:00.062) 0:02:12.991 ***** 2021-02-11T00:25:01.779 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-02-11T00:25:01.780 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 00:22:23 UTC", "ActiveEnterTimestampMonotonic": "27602124", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target systemd-tmpfiles-setup.service network.target -.mount systemd-journald.socket basic.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 00:22:16 UTC", "AssertTimestampMonotonic": "21060080", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 00:22:16 UTC", "ConditionTimestampMonotonic": "21060080", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-02-11T00:25:01.781 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1637", "ExecMainStartTimestamp": "Thu 2021-02-11 00:22:23 UTC", "ExecMainStartTimestampMonotonic": "27602110", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 00:22:16 UTC", "InactiveExitTimestampMonotonic": "21060982", "InvocationID": "0c9c22bf331648c6b47f6a202a2386ad", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-02-11T00:25:01.782 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1637", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 00:22:23 UTC", "S 2021-02-11T00:25:01.784 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "27602124", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 00:22:23 UTC", "WatchdogTimestampMonotonic": "27602122", "WatchdogUSec": "0"}} 2021-02-11T00:25:01.821 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:01 +0000 (0:00:00.256) 0:02:13.247 ***** 2021-02-11T00:25:01.847 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T00:25:01.849 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-02-11T00:25:01.884 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:01 +0000 (0:00:00.063) 0:02:13.311 ***** 2021-02-11T00:25:01.911 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:01.947 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:01 +0000 (0:00:00.063) 0:02:13.374 ***** 2021-02-11T00:25:01.974 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-02-11T00:25:01.976 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:02.009 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:02 +0000 (0:00:00.061) 0:02:13.436 ***** 2021-02-11T00:25:02.047 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-02-11T00:25:02.048 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"cache_update_time": 1613002741, "cache_updated": false, "changed": false} 2021-02-11T00:25:02.084 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:02 +0000 (0:00:00.074) 0:02:13.511 ***** 2021-02-11T00:25:02.112 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-02-11T00:25:02.114 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:02.158 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:02 +0000 (0:00:00.073) 0:02:13.584 ***** 2021-02-11T00:25:02.478 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-02-11T00:25:02.479 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-02-11T00:25:02.516 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:02 +0000 (0:00:00.358) 0:02:13.943 ***** 2021-02-11T00:25:02.543 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:02.579 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:02 +0000 (0:00:00.063) 0:02:14.006 ***** 2021-02-11T00:25:02.795 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-02-11T00:25:02.796 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-02-11 00:22:18 UTC", "ActiveEnterTimestampMonotonic": "25024910", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service basic.target -.mount system.slice network.target systemd-journald.socket sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-02-11 00:22:12 UTC", "AssertTimestampMonotonic": "18448036", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-02-11 00:22:12 UTC", "ConditionTimestampMonotonic": "18448036", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-02-11T00:25:02.798 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1772", "ExecMainStartTimestamp": "Thu 2021-02-11 00:22:18 UTC", "ExecMainStartTimestampMonotonic": "25024896", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-02-11 00:22:12 UTC", "InactiveExitTimestampMonotonic": "18448764", "InvocationID": "186a61336d8a4c6794c7b471be20de03", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-02-11T00:25:02.799 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1772", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 00:22:18 UTC", "S 2021-02-11T00:25:02.800 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "25024910", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-02-11 00:22:18 UTC", "WatchdogTimestampMonotonic": "25024908", "WatchdogUSec": "0"}} 2021-02-11T00:25:02.832 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:02 +0000 (0:00:00.252) 0:02:14.259 ***** 2021-02-11T00:25:02.859 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T00:25:02.861 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} TASK [testnode : Install python-pip on apt based systems.] ********************* ok: [smithi095.front.sepia.ceph.com] => {"cache_update_time": 1613002785, "cache_updated": false, "changed": false} 2021-02-11T00:25:02.895 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:02 +0000 (0:00:00.062) 0:02:14.321 ***** 2021-02-11T00:25:02.930 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:02 +0000 (0:00:00.033) 0:02:14.355 ***** 2021-02-11T00:25:05.336 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 00:25:02 +0000 (0:00:00.052) 0:02:14.407 ***** TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 00:25:03 +0000 (0:00:00.043) 0:02:14.451 ***** TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 00:25:03 +0000 (0:00:00.047) 0:02:14.498 ***** TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} Thursday 11 February 2021 00:25:03 +0000 (0:00:00.128) 0:02:14.627 ***** TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} Thursday 11 February 2021 00:25:03 +0000 (0:00:00.379) 0:02:15.007 ***** TASK [testnode : Install python-pip on apt based systems.] ********************* ok: [smithi057.front.sepia.ceph.com] => {"cache_update_time": 1613002741, "cache_updated": false, "changed": false} Thursday 11 February 2021 00:25:04 +0000 (0:00:00.493) 0:02:15.500 ***** TASK [testnode : Install python-pip on z 2021-02-11T00:25:05.338 INFO:teuthology.task.ansible.out:ypper based systems.] ****************** skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 11 February 2021 00:25:04 +0000 (0:00:00.062) 0:02:15.562 ***** TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} Thursday 11 February 2021 00:25:04 +0000 (0:00:00.166) 0:02:15.729 ***** TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} TASK [testnode : Install packages via pip] ************************************* ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} Thursday 1 2021-02-11T00:25:05.339 INFO:teuthology.task.ansible.out:1 February 2021 00:25:04 +0000 (0:00:00.366) 0:02:16.096 ***** Thursday 11 February 2021 00:25:04 +0000 (0:00:00.036) 0:02:16.132 ***** TASK [testnode : Purge resolvconf] ********************************************* ok: [smithi095.front.sepia.ceph.com] => {"changed": false} Thursday 11 February 2021 00:25:05 +0000 (0:00:00.550) 0:02:16.683 ***** TASK [testnode : Set interface] ************************************************ ok: [smithi095.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} Thursday 11 February 2021 00:25:05 +0000 (0:00:00.061) 0:02:16.745 ***** 2021-02-11T00:25:05.439 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-02-11T00:25:05.440 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-02-11T00:25:05.475 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:05 +0000 (0:00:00.156) 0:02:16.902 ***** 2021-02-11T00:25:05.483 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-02-11T00:25:05.483 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1613002708.163546, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1613002708.159546, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 746, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1613002708.159546, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3375002756", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-02-11T00:25:05.520 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:05 +0000 (0:00:00.044) 0:02:16.947 ***** 2021-02-11T00:25:05.900 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-02-11T00:25:05.902 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-02-11T00:25:05.937 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:05 +0000 (0:00:00.416) 0:02:17.363 ***** 2021-02-11T00:25:05.963 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-02-11T00:25:05.964 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-02-11T00:25:05.998 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:05 +0000 (0:00:00.061) 0:02:17.424 ***** 2021-02-11T00:25:06.003 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-02-11T00:25:06.004 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false} 2021-02-11T00:25:06.039 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:06 +0000 (0:00:00.041) 0:02:17.466 ***** 2021-02-11T00:25:06.049 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-02-11T00:25:06.050 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:06.086 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:06 +0000 (0:00:00.046) 0:02:17.513 ***** 2021-02-11T00:25:06.091 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-02-11T00:25:06.092 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-02-11T00:25:06.129 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:06 +0000 (0:00:00.042) 0:02:17.555 ***** 2021-02-11T00:25:06.222 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-02-11T00:25:06.224 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-02-11T00:25:06.256 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:06 +0000 (0:00:00.127) 0:02:17.682 ***** 2021-02-11T00:25:06.283 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-02-11T00:25:06.283 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1613002705.9553268, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1613002705.9513268, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 746, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1613002705.9513268, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "3375002756", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-02-11T00:25:06.320 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:06 +0000 (0:00:00.064) 0:02:17.747 ***** 2021-02-11T00:25:06.389 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-02-11T00:25:06.390 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-02-11T00:25:06.426 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:06 +0000 (0:00:00.105) 0:02:17.853 ***** 2021-02-11T00:25:06.452 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:06.453 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-02-11T00:25:06.453 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-02-11T00:25:06.487 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:06 +0000 (0:00:00.061) 0:02:17.914 ***** 2021-02-11T00:25:06.520 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:06 +0000 (0:00:00.033) 0:02:17.947 ***** 2021-02-11T00:25:06.526 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-02-11T00:25:06.527 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-02-11T00:25:06.564 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:06 +0000 (0:00:00.043) 0:02:17.991 ***** 2021-02-11T00:25:06.571 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-02-11T00:25:06.572 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-02-11T00:25:06.573 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi095.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-02-11T00:25:06.573 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-02-11T00:25:06.584 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-02-11T00:25:06.620 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:06 +0000 (0:00:00.056) 0:02:18.047 ***** 2021-02-11T00:25:06.626 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-02-11T00:25:06.627 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:06.666 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:06 +0000 (0:00:00.045) 0:02:18.093 ***** 2021-02-11T00:25:06.674 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-02-11T00:25:06.674 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:06.676 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:06.676 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:06.678 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi095.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:06.716 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:06 +0000 (0:00:00.049) 0:02:18.143 ***** 2021-02-11T00:25:06.742 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T00:25:06.743 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:06.780 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:06 +0000 (0:00:00.063) 0:02:18.206 ***** 2021-02-11T00:25:06.806 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-02-11T00:25:06.808 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi057.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-02-11T00:25:06.842 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:06 +0000 (0:00:00.062) 0:02:18.269 ***** 2021-02-11T00:25:06.876 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:06 +0000 (0:00:00.033) 0:02:18.303 ***** 2021-02-11T00:25:06.905 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-02-11T00:25:06.906 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:06.947 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:06 +0000 (0:00:00.070) 0:02:18.373 ***** 2021-02-11T00:25:06.975 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-02-11T00:25:06.975 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-02-11T00:25:06.976 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.031 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:07 +0000 (0:00:00.084) 0:02:18.458 ***** 2021-02-11T00:25:07.067 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:07 +0000 (0:00:00.035) 0:02:18.494 ***** 2021-02-11T00:25:07.074 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-02-11T00:25:07.076 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.109 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:07 +0000 (0:00:00.042) 0:02:18.536 ***** 2021-02-11T00:25:07.116 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-02-11T00:25:07.116 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.155 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:07 +0000 (0:00:00.046) 0:02:18.582 ***** 2021-02-11T00:25:07.162 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-02-11T00:25:07.162 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.163 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi057.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.165 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} skipping: [smithi057.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.200 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:07 +0000 (0:00:00.044) 0:02:18.626 ***** 2021-02-11T00:25:07.206 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-11T00:25:07.207 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.244 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:07 +0000 (0:00:00.044) 0:02:18.671 ***** 2021-02-11T00:25:07.250 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-02-11T00:25:07.251 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.252 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi057.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.254 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.264 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.300 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:07 +0000 (0:00:00.056) 0:02:18.727 ***** 2021-02-11T00:25:07.306 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-02-11T00:25:07.307 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.343 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:07 +0000 (0:00:00.042) 0:02:18.769 ***** 2021-02-11T00:25:07.348 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-11T00:25:07.350 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.387 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:07 +0000 (0:00:00.044) 0:02:18.814 ***** 2021-02-11T00:25:07.394 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-11T00:25:07.394 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.430 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:07 +0000 (0:00:00.042) 0:02:18.857 ***** 2021-02-11T00:25:07.436 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-02-11T00:25:07.438 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.473 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:07 +0000 (0:00:00.042) 0:02:18.900 ***** 2021-02-11T00:25:07.500 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.543 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:07 +0000 (0:00:00.070) 0:02:18.970 ***** 2021-02-11T00:25:07.581 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-11T00:25:07.583 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011061", "end": "2021-02-11 00:25:07.576870", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-11 00:25:07.565809", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:25:07.696 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005981", "end": "2021-02-11 00:25:07.699041", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-11 00:25:07.693060", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:25:07.696 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-02-11T00:25:07.697 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.846 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:07 +0000 (0:00:00.302) 0:02:19.273 ***** 2021-02-11T00:25:07.853 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-11T00:25:07.854 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.003640", "end": "2021-02-11 00:25:07.810761", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-11 00:25:07.807121", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-02-11T00:25:07.876 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.913 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:07 +0000 (0:00:00.066) 0:02:19.340 ***** 2021-02-11T00:25:07.940 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:07.977 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:07 +0000 (0:00:00.063) 0:02:19.403 ***** 2021-02-11T00:25:08.014 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-02-11T00:25:08.016 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:08.052 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:08 +0000 (0:00:00.075) 0:02:19.479 ***** 2021-02-11T00:25:08.059 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-11T00:25:08.059 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.007305", "end": "2021-02-11 00:25:08.035132", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-11 00:25:08.027827", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:25:08.098 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:08 +0000 (0:00:00.045) 0:02:19.524 ***** 2021-02-11T00:25:08.105 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-11T00:25:08.106 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:08.140 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:08 +0000 (0:00:00.042) 0:02:19.567 ***** 2021-02-11T00:25:08.147 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-02-11T00:25:08.184 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:08 +0000 (0:00:00.043) 0:02:19.611 ***** 2021-02-11T00:25:08.295 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-11T00:25:08.296 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011086", "end": "2021-02-11 00:25:08.431700", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-11 00:25:08.420614", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:25:08.317 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-02-11T00:25:08.319 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-02-11T00:25:08.422 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-11T00:25:08.423 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005361", "end": "2021-02-11 00:25:08.562174", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-11 00:25:08.556813", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:25:08.529 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005348", "end": "2021-02-11 00:25:08.677055", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-11 00:25:08.671707", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-02-11T00:25:08.695 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.046600", "end": "2021-02-11 00:25:08.829185", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-11 00:25:08.782585", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:25:08.730 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:08 +0000 (0:00:00.546) 0:02:20.157 ***** 2021-02-11T00:25:08.767 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-02-11T00:25:08.804 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:08 +0000 (0:00:00.074) 0:02:20.231 ***** 2021-02-11T00:25:08.945 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-02-11T00:25:08.947 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-02-11T00:25:09.051 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-02-11T00:25:09.084 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:09 +0000 (0:00:00.279) 0:02:20.511 ***** 2021-02-11T00:25:09.120 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:09 +0000 (0:00:00.035) 0:02:20.546 ***** 2021-02-11T00:25:09.739 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi057.front.sepia.ceph.com] 2021-02-11T00:25:09.817 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:09 +0000 (0:00:00.697) 0:02:21.244 ***** 2021-02-11T00:25:09.858 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T00:25:09.860 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-02-11T00:25:09.894 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:09 +0000 (0:00:00.076) 0:02:21.320 ***** 2021-02-11T00:25:09.921 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T00:25:09.923 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:09.925 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-02-11T00:25:09.927 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] 2021-02-11T00:25:09.966 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:09 +0000 (0:00:00.072) 0:02:21.393 ***** 2021-02-11T00:25:10.002 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-02-11T00:25:10.004 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-02-11T00:25:10.038 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:10 +0000 (0:00:00.072) 0:02:21.465 ***** 2021-02-11T00:25:10.074 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:10 +0000 (0:00:00.035) 0:02:21.501 ***** 2021-02-11T00:25:10.101 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T00:25:10.102 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-02-11T00:25:10.138 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:10 +0000 (0:00:00.064) 0:02:21.565 ***** 2021-02-11T00:25:10.165 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T00:25:10.166 INFO:teuthology.task.ansible.out:skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:10.201 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:10 +0000 (0:00:00.063) 0:02:21.628 ***** 2021-02-11T00:25:10.249 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-02-11T00:25:10.251 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-02-11T00:25:10.286 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:10 +0000 (0:00:00.084) 0:02:21.713 ***** 2021-02-11T00:25:10.963 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-11T00:25:10.964 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"cache_update_time": 1613002741, "cache_updated": false, "changed": false} 2021-02-11T00:25:11.005 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:11 +0000 (0:00:00.719) 0:02:22.432 ***** 2021-02-11T00:25:11.032 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T00:25:11.033 INFO:teuthology.task.ansible.out:skipping: [smithi057.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:11.069 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:11 +0000 (0:00:00.063) 0:02:22.495 ***** 2021-02-11T00:25:11.095 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-11T00:25:11.096 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-02-11T00:25:11.132 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:11 +0000 (0:00:00.063) 0:02:22.558 ***** 2021-02-11T00:25:11.200 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-11T00:25:11.201 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"cache_update_time": 1613002785, "cache_updated": false, "changed": false} 2021-02-11T00:25:11.240 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:11 +0000 (0:00:00.107) 0:02:22.666 ***** 2021-02-11T00:25:11.267 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi095.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-11T00:25:11.305 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:11 +0000 (0:00:00.065) 0:02:22.732 ***** 2021-02-11T00:25:11.311 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-02-11T00:25:11.312 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T00:25:11.333 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi095.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-02-11T00:25:11.370 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:11 +0000 (0:00:00.065) 0:02:22.797 ***** 2021-02-11T00:25:11.386 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-02-11T00:25:11.386 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T00:25:11.428 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:11 +0000 (0:00:00.058) 0:02:22.855 ***** 2021-02-11T00:25:11.508 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T00:25:11.615 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-02-11T00:25:11.654 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:11 +0000 (0:00:00.225) 0:02:23.081 ***** 2021-02-11T00:25:11.773 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-02-11T00:25:11.775 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613003111.4652731-122155197635034/source", "state": "file", "uid": 0} 2021-02-11T00:25:11.972 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613003111.6902664-271286935349486/source", "state": "file", "uid": 0} 2021-02-11T00:25:11.973 INFO:teuthology.task.ansible.out: 2021-02-11T00:25:12.087 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613003111.7803154-114742457772014/source", "state": "file", "uid": 0} 2021-02-11T00:25:12.128 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:12 +0000 (0:00:00.473) 0:02:23.554 ***** 2021-02-11T00:25:12.280 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613003111.984245-80978014179185/source", "state": "file", "uid": 0} 2021-02-11T00:25:12.317 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:12 +0000 (0:00:00.188) 0:02:23.743 ***** 2021-02-11T00:25:15.774 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-02-11T00:25:15.776 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-eIC8dR-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-eIC8dR-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-02-11T00:25:15.777 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-02-11T00:25:15.813 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:15 +0000 (0:00:03.496) 0:02:27.240 ***** 2021-02-11T00:25:15.830 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-bQZLxh-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-bQZLxh-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-02-11T00:25:15.832 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-02-11T00:25:15.871 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:15 +0000 (0:00:00.057) 0:02:27.298 ***** 2021-02-11T00:25:15.952 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-02-11T00:25:15.953 INFO:teuthology.task.ansible.out:ok: [smithi095.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-11T00:25:15.993 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:15 +0000 (0:00:00.122) 0:02:27.420 ***** 2021-02-11T00:25:16.021 INFO:teuthology.task.ansible.out:ok: [smithi057.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-11T00:25:16.066 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:16 +0000 (0:00:00.072) 0:02:27.492 ***** 2021-02-11T00:25:16.321 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-02-11T00:25:16.323 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613003116.0260444-257084157441335/source", "state": "file", "uid": 0} 2021-02-11T00:25:16.360 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:16 +0000 (0:00:00.294) 0:02:27.786 ***** 2021-02-11T00:25:16.418 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613003116.103028-44864317360624/source", "state": "file", "uid": 0} 2021-02-11T00:25:16.457 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:16 +0000 (0:00:00.096) 0:02:27.883 ***** 2021-02-11T00:25:16.599 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-02-11T00:25:16.601 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.109093", "end": "2021-02-11 00:25:16.603957", "rc": 0, "start": "2021-02-11 00:25:16.494864", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:25:16.636 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:16 +0000 (0:00:00.179) 0:02:28.063 ***** 2021-02-11T00:25:16.716 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.123803", "end": "2021-02-11 00:25:16.860691", "rc": 0, "start": "2021-02-11 00:25:16.736888", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-11T00:25:16.753 INFO:teuthology.task.ansible.out:Thursday 11 February 2021 00:25:16 +0000 (0:00:00.116) 0:02:28.179 ***** 2021-02-11T00:25:19.920 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-02-11T00:25:19.921 INFO:teuthology.task.ansible.out:changed: [smithi095.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "sysinit.target systemd-journald.socket firewalld.service containerd.service basic.target system.slice docker.socket network-online.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-02-11T00:25:19.922 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "12802 2021-02-11T00:25:19.923 INFO:teuthology.task.ansible.out:9", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "docker.socket sysinit.target system.slice", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 00:24:15 UTC", "StateChangeTimestampMonotonic": "139891006", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "S 2021-02-11T00:25:19.925 INFO:teuthology.task.ansible.out:uccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-online.target containerd.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-11T00:25:20.112 INFO:teuthology.task.ansible.out:changed: [smithi057.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "systemd-journald.socket docker.socket basic.target sysinit.target network-online.target firewalld.service containerd.service system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-02-11T00:25:20.113 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "12805 2021-02-11T00:25:20.115 INFO:teuthology.task.ansible.out:3", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target docker.socket system.slice", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-02-11 00:24:17 UTC", "StateChangeTimestampMonotonic": "143782763", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "S 2021-02-11T00:25:20.116 INFO:teuthology.task.ansible.out:uccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-11T00:25:20.213 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-02-11T00:25:20.217 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-02-11T00:25:20.221 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* 2021-02-11T00:25:20.222 INFO:teuthology.task.ansible.out:smithi057.front.sepia.ceph.com : ok=126 changed=38 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 smithi095.front.sepia.ceph.com : ok=126 changed=38 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 Thursday 11 February 2021 00:25:20 +0000 (0:00:03.468) 0:02:31.648 ***** =============================================================================== users : Create all admin users with sudo access. ----------------------- 18.68s users : Update authorized_keys using the keys repo --------------------- 15.26s testnode : Zap all non-root disks --------------------------------------- 7.59s testnode : Ensure packages are not present. ----------------------------- 7.59s testnode : Install packages --------------------------------------------- 7.51s container-host : Install registries-conf-ctl ---------------------------- 3.50s container-host : Restart docker service --------------------------------- 3.47s users : Remove revoked users -------------------------------------------- 2.95s Gathering Facts --------------------------------------------------------- 1.81s users : Update authorized_keys for each user with literal keys ---------- 1.57s testnode : Update apt cache. -------------------------------------------- 1.47s testnode : Install python-apt ------------------------------------------- 0.94s common : Set the hardware clock ----------------------------------------- 0.87s testnode : Create logical volume(s) ------------------------------------- 0.86s common : Install Kerberos Packages (Debian) ----------------------------- 0.85s common : Make sure smartmontools is installed --------------------------- 0.78s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.76s testnode : Blow away lingering OSD data and FSIDs ----------------------- 0.74s Gathering Facts --------------------------------------------------------- 0.73s container-host : Install container packages ----- 2021-02-11T00:25:20.223 INFO:teuthology.task.ansible.out:------------------------ 0.72s 2021-02-11T00:25:20.561 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi057.front.sepia.ceph.com'), Remote(name='ubuntu@smithi095.front.sepia.ceph.com')] 2021-02-11T00:25:20.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T00:25:20.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T00:25:20.642 DEBUG:teuthology.orchestra.run.smithi057:> true 2021-02-11T00:25:20.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-11T00:25:20.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi095.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-11T00:25:20.926 DEBUG:teuthology.orchestra.run.smithi095:> true 2021-02-11T00:25:21.127 INFO:teuthology.run_tasks:Running task clock... 2021-02-11T00:25:21.139 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-02-11T00:25:21.139 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-11T00:25:21.141 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-11T00:25:21.200 INFO:teuthology.orchestra.run.smithi057.stdout:11 Feb 00:25:21 ntpd[8347]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-02-11T00:25:21.200 INFO:teuthology.orchestra.run.smithi057.stdout:11 Feb 00:25:21 ntpd[8347]: Command line: ntpd -gq 2021-02-11T00:25:21.201 INFO:teuthology.orchestra.run.smithi057.stdout:11 Feb 00:25:21 ntpd[8347]: proto: precision = 0.055 usec (-24) 2021-02-11T00:25:21.201 INFO:teuthology.orchestra.run.smithi057.stdout:11 Feb 00:25:21 ntpd[8347]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-11T00:25:21.201 INFO:teuthology.orchestra.run.smithi057.stdout:11 Feb 00:25:21 ntpd[8347]: restrict ::: KOD does nothing without LIMITED. 2021-02-11T00:25:21.201 INFO:teuthology.orchestra.run.smithi057.stdout:11 Feb 00:25:21 ntpd[8347]: Listen and drop on 0 v6wildcard [::]:123 2021-02-11T00:25:21.201 INFO:teuthology.orchestra.run.smithi057.stdout:11 Feb 00:25:21 ntpd[8347]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-02-11T00:25:21.201 INFO:teuthology.orchestra.run.smithi057.stdout:11 Feb 00:25:21 ntpd[8347]: Listen normally on 2 lo 127.0.0.1:123 2021-02-11T00:25:21.201 INFO:teuthology.orchestra.run.smithi057.stdout:11 Feb 00:25:21 ntpd[8347]: Listen normally on 3 ens1f0 172.21.15.57:123 2021-02-11T00:25:21.202 INFO:teuthology.orchestra.run.smithi057.stdout:11 Feb 00:25:21 ntpd[8347]: Listen normally on 4 lo [::1]:123 2021-02-11T00:25:21.202 INFO:teuthology.orchestra.run.smithi057.stdout:11 Feb 00:25:21 ntpd[8347]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1562%3]:123 2021-02-11T00:25:21.202 INFO:teuthology.orchestra.run.smithi057.stdout:11 Feb 00:25:21 ntpd[8347]: Listening on routing socket on fd #22 for interface updates 2021-02-11T00:25:21.202 INFO:teuthology.orchestra.run.smithi057.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-11T00:25:21.202 INFO:teuthology.orchestra.run.smithi057.stderr:restrict ::: KOD does nothing without LIMITED. 2021-02-11T00:25:22.202 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:22 ntpd[8347]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:22.202 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:22 ntpd[8347]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:22.202 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:22 ntpd[8347]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:22.202 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:22 ntpd[8347]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:22.202 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:22 ntpd[8347]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:22.202 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:22 ntpd[8347]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:22.202 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:22 ntpd[8347]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:22.202 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:22 ntpd[8347]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:22.202 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:22 ntpd[8347]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:22.203 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:22 ntpd[8347]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:22.203 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:22 ntpd[8347]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:22.203 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:22 ntpd[8347]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:24.202 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:24 ntpd[8347]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:24.203 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:24 ntpd[8347]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:24.203 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:24 ntpd[8347]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:24.203 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:24 ntpd[8347]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:24.203 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:24 ntpd[8347]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:24.203 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:24 ntpd[8347]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:24.203 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:24 ntpd[8347]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:24.203 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:24 ntpd[8347]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:24.203 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:24 ntpd[8347]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:24.203 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:24 ntpd[8347]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:24.203 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:24 ntpd[8347]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:24.203 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:24 ntpd[8347]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:26.204 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:26 ntpd[8347]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:26.204 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:26 ntpd[8347]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:26.204 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:26 ntpd[8347]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:26.204 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:26 ntpd[8347]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:26.204 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:26 ntpd[8347]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:26.204 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:26 ntpd[8347]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:26.204 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:26 ntpd[8347]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:26.205 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:26 ntpd[8347]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:26.205 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:26 ntpd[8347]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:26.205 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:26 ntpd[8347]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:26.205 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:26 ntpd[8347]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:26.205 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:26 ntpd[8347]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:26.205 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:26 ntpd[8347]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:26.205 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:26 ntpd[8347]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:26.205 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:26 ntpd[8347]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:26.205 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:26 ntpd[8347]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:28.204 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:28 ntpd[8347]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:28.204 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:28 ntpd[8347]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:28.204 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:28 ntpd[8347]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:28.204 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:28 ntpd[8347]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:28.205 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:28 ntpd[8347]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-02-11T00:25:28.205 INFO:teuthology.orchestra.run.smithi057.stderr:11 Feb 00:25:28 ntpd[8347]: can't open /var/log/ntpstats/loopstats.20210211: Permission denied 2021-02-11T00:25:28.205 INFO:teuthology.orchestra.run.smithi057.stdout:11 Feb 00:25:28 ntpd[8347]: ntpd: time set -0.158750 s 2021-02-11T00:25:28.205 INFO:teuthology.orchestra.run.smithi057.stdout:ntpd: time set -0.158750s 2021-02-11T00:25:28.278 INFO:teuthology.orchestra.run.smithi057.stdout: remote refid st t when poll reach delay offset jitter 2021-02-11T00:25:28.278 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================== 2021-02-11T00:25:28.278 INFO:teuthology.orchestra.run.smithi057.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T00:25:28.278 INFO:teuthology.orchestra.run.smithi057.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T00:25:28.278 INFO:teuthology.orchestra.run.smithi057.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T00:25:28.278 INFO:teuthology.orchestra.run.smithi057.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T00:25:28.279 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-11T00:25:28.280 DEBUG:teuthology.orchestra.run.smithi095:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-11T00:25:28.340 INFO:teuthology.orchestra.run.smithi095.stdout:11 Feb 00:25:28 ntpd[8202]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-02-11T00:25:28.340 INFO:teuthology.orchestra.run.smithi095.stdout:11 Feb 00:25:28 ntpd[8202]: Command line: ntpd -gq 2021-02-11T00:25:28.341 INFO:teuthology.orchestra.run.smithi095.stdout:11 Feb 00:25:28 ntpd[8202]: proto: precision = 0.126 usec (-23) 2021-02-11T00:25:28.342 INFO:teuthology.orchestra.run.smithi095.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-11T00:25:28.342 INFO:teuthology.orchestra.run.smithi095.stderr:restrict ::: KOD does nothing without LIMITED. 2021-02-11T00:25:28.342 INFO:teuthology.orchestra.run.smithi095.stdout:11 Feb 00:25:28 ntpd[8202]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-02-11T00:25:28.342 INFO:teuthology.orchestra.run.smithi095.stdout:11 Feb 00:25:28 ntpd[8202]: restrict ::: KOD does nothing without LIMITED. 2021-02-11T00:25:28.342 INFO:teuthology.orchestra.run.smithi095.stdout:11 Feb 00:25:28 ntpd[8202]: Listen and drop on 0 v6wildcard [::]:123 2021-02-11T00:25:28.342 INFO:teuthology.orchestra.run.smithi095.stdout:11 Feb 00:25:28 ntpd[8202]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-02-11T00:25:28.343 INFO:teuthology.orchestra.run.smithi095.stdout:11 Feb 00:25:28 ntpd[8202]: Listen normally on 2 lo 127.0.0.1:123 2021-02-11T00:25:28.343 INFO:teuthology.orchestra.run.smithi095.stdout:11 Feb 00:25:28 ntpd[8202]: Listen normally on 3 enp3s0f1 172.21.15.95:123 2021-02-11T00:25:28.343 INFO:teuthology.orchestra.run.smithi095.stdout:11 Feb 00:25:28 ntpd[8202]: Listen normally on 4 lo [::1]:123 2021-02-11T00:25:28.344 INFO:teuthology.orchestra.run.smithi095.stdout:11 Feb 00:25:28 ntpd[8202]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d5f9%5]:123 2021-02-11T00:25:28.344 INFO:teuthology.orchestra.run.smithi095.stdout:11 Feb 00:25:28 ntpd[8202]: Listening on routing socket on fd #22 for interface updates 2021-02-11T00:25:30.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:30 ntpd[8202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:30.342 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:30 ntpd[8202]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:30.342 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:30 ntpd[8202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:30.342 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:30 ntpd[8202]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:30.342 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:30 ntpd[8202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:30.342 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:30 ntpd[8202]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:30.342 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:30 ntpd[8202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:30.342 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:30 ntpd[8202]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:30.342 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:30 ntpd[8202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:30.342 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:30 ntpd[8202]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:30.342 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:30 ntpd[8202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:30.342 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:30 ntpd[8202]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:31.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:31 ntpd[8202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:31.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:31 ntpd[8202]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:31.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:31 ntpd[8202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:31.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:31 ntpd[8202]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:32.342 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:32 ntpd[8202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:32.343 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:32 ntpd[8202]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:32.343 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:32 ntpd[8202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:32.343 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:32 ntpd[8202]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:32.343 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:32 ntpd[8202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:32.343 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:32 ntpd[8202]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:32.344 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:32 ntpd[8202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:32.344 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:32 ntpd[8202]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:32.344 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:32 ntpd[8202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:32.344 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:32 ntpd[8202]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:32.344 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:32 ntpd[8202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:32.344 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:32 ntpd[8202]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:33.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:33 ntpd[8202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:33.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:33 ntpd[8202]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:33.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:33 ntpd[8202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:33.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:33 ntpd[8202]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:34.342 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:34 ntpd[8202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:34.342 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:34 ntpd[8202]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:34.343 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:34 ntpd[8202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:34.343 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:34 ntpd[8202]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:34.343 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:34 ntpd[8202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:34.343 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:34 ntpd[8202]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:34.343 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:34 ntpd[8202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:34.343 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:34 ntpd[8202]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:34.343 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:34 ntpd[8202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:34.343 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:34 ntpd[8202]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:34.344 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:34 ntpd[8202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:34.344 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:34 ntpd[8202]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:35.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:35 ntpd[8202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:35.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:35 ntpd[8202]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:35.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:35 ntpd[8202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:35.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:35 ntpd[8202]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:36.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:36 ntpd[8202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-02-11T00:25:36.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:36 ntpd[8202]: can't open /var/log/ntpstats/rawstats.20210211: Permission denied 2021-02-11T00:25:36.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:36 ntpd[8202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-02-11T00:25:36.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:36 ntpd[8202]: can't open /var/log/ntpstats/peerstats.20210211: Permission denied 2021-02-11T00:25:36.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:36 ntpd[8202]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-02-11T00:25:36.341 INFO:teuthology.orchestra.run.smithi095.stderr:11 Feb 00:25:36 ntpd[8202]: can't open /var/log/ntpstats/loopstats.20210211: Permission denied 2021-02-11T00:25:36.342 INFO:teuthology.orchestra.run.smithi095.stdout:11 Feb 00:25:36 ntpd[8202]: ntpd: time slew -0.021442 s 2021-02-11T00:25:36.342 INFO:teuthology.orchestra.run.smithi095.stdout:ntpd: time slew -0.021442s 2021-02-11T00:25:36.397 INFO:teuthology.orchestra.run.smithi095.stdout: remote refid st t when poll reach delay offset jitter 2021-02-11T00:25:36.397 INFO:teuthology.orchestra.run.smithi095.stdout:============================================================================== 2021-02-11T00:25:36.397 INFO:teuthology.orchestra.run.smithi095.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T00:25:36.397 INFO:teuthology.orchestra.run.smithi095.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T00:25:36.397 INFO:teuthology.orchestra.run.smithi095.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T00:25:36.397 INFO:teuthology.orchestra.run.smithi095.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-11T00:25:36.397 INFO:teuthology.run_tasks:Running task install... 2021-02-11T00:25:36.404 DEBUG:teuthology.task.install:project ceph 2021-02-11T00:25:36.405 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '597c1051df9f309aa7d6c6db46dcbc2c3982c6e3'}} 2021-02-11T00:25:36.405 DEBUG:teuthology.task.install:config {'sha1': '597c1051df9f309aa7d6c6db46dcbc2c3982c6e3'} 2021-02-11T00:25:36.405 INFO:teuthology.task.install:Using flavor: basic 2021-02-11T00:25:36.414 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-02-11T00:25:36.414 INFO:teuthology.task.install:extra packages: [] 2021-02-11T00:25:36.414 DEBUG:teuthology.orchestra.run.smithi057:> sudo lsb_release -is 2021-02-11T00:25:36.469 INFO:teuthology.orchestra.run.smithi057.stdout:Ubuntu 2021-02-11T00:25:36.474 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T00:25:36.474 DEBUG:teuthology.orchestra.run.smithi095:> sudo lsb_release -is 2021-02-11T00:25:36.475 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt-key list | grep Ceph 2021-02-11T00:25:36.525 INFO:teuthology.orchestra.run.smithi095.stdout:Ubuntu 2021-02-11T00:25:36.530 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T00:25:36.530 DEBUG:teuthology.orchestra.run.smithi095:> sudo apt-key list | grep Ceph 2021-02-11T00:25:36.576 INFO:teuthology.orchestra.run.smithi057.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-02-11T00:25:36.621 INFO:teuthology.orchestra.run.smithi095.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-02-11T00:25:36.632 INFO:teuthology.orchestra.run.smithi057.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-02-11T00:25:36.632 INFO:teuthology.orchestra.run.smithi057.stdout:uid [ unknown] Ceph.com (release key) 2021-02-11T00:25:36.633 DEBUG:teuthology.orchestra.run.smithi057:> sudo lsb_release -is 2021-02-11T00:25:36.677 INFO:teuthology.orchestra.run.smithi095.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-02-11T00:25:36.678 INFO:teuthology.orchestra.run.smithi095.stdout:uid [ unknown] Ceph.com (release key) 2021-02-11T00:25:36.679 DEBUG:teuthology.orchestra.run.smithi095:> sudo lsb_release -is 2021-02-11T00:25:36.683 INFO:teuthology.orchestra.run.smithi057.stdout:Ubuntu 2021-02-11T00:25:36.689 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T00:25:36.689 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-02-11T00:25:36.689 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=597c1051df9f309aa7d6c6db46dcbc2c3982c6e3 2021-02-11T00:25:36.732 INFO:teuthology.orchestra.run.smithi095.stdout:Ubuntu 2021-02-11T00:25:36.737 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T00:25:36.737 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-02-11T00:25:36.737 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=597c1051df9f309aa7d6c6db46dcbc2c3982c6e3 2021-02-11T00:25:36.865 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default/ 2021-02-11T00:25:36.865 INFO:teuthology.task.install.deb:Package version is 17.0.0-691-g597c1051-1bionic 2021-02-11T00:25:36.900 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default/ 2021-02-11T00:25:36.901 INFO:teuthology.task.install.deb:Package version is 17.0.0-691-g597c1051-1bionic 2021-02-11T00:25:36.990 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:25:36.991 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-02-11T00:25:37.006 DEBUG:teuthology.orchestra.run.smithi057:> sudo apt-get update 2021-02-11T00:25:37.022 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:25:37.022 DEBUG:teuthology.orchestra.run.smithi095:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-02-11T00:25:37.037 DEBUG:teuthology.orchestra.run.smithi095:> sudo apt-get update 2021-02-11T00:25:37.185 INFO:teuthology.orchestra.run.smithi057.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-02-11T00:25:37.259 INFO:teuthology.orchestra.run.smithi057.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic InRelease 2021-02-11T00:25:37.284 INFO:teuthology.orchestra.run.smithi057.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-02-11T00:25:37.286 INFO:teuthology.orchestra.run.smithi095.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic InRelease 2021-02-11T00:25:37.303 INFO:teuthology.orchestra.run.smithi057.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T00:25:37.309 INFO:teuthology.orchestra.run.smithi095.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-02-11T00:25:37.310 INFO:teuthology.orchestra.run.smithi057.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic Release.gpg 2021-02-11T00:25:37.334 INFO:teuthology.orchestra.run.smithi057.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.6 kB] 2021-02-11T00:25:37.342 INFO:teuthology.orchestra.run.smithi095.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-02-11T00:25:37.343 INFO:teuthology.orchestra.run.smithi095.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-02-11T00:25:37.343 INFO:teuthology.orchestra.run.smithi095.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic Release.gpg 2021-02-11T00:25:37.367 INFO:teuthology.orchestra.run.smithi095.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.6 kB] 2021-02-11T00:25:37.376 INFO:teuthology.orchestra.run.smithi057.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,105 B] 2021-02-11T00:25:37.384 INFO:teuthology.orchestra.run.smithi057.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-02-11T00:25:37.406 INFO:teuthology.orchestra.run.smithi095.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,105 B] 2021-02-11T00:25:37.408 INFO:teuthology.orchestra.run.smithi095.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-02-11T00:25:37.686 INFO:teuthology.orchestra.run.smithi057.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-02-11T00:25:37.708 INFO:teuthology.orchestra.run.smithi095.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-02-11T00:25:37.950 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 279 kB in 1s (378 kB/s) 2021-02-11T00:25:38.020 INFO:teuthology.orchestra.run.smithi095.stdout:Fetched 279 kB in 1s (364 kB/s) 2021-02-11T00:25:39.237 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2021-02-11T00:25:39.285 DEBUG:teuthology.orchestra.run.smithi057:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-691-g597c1051-1bionic cephadm=17.0.0-691-g597c1051-1bionic ceph-mds=17.0.0-691-g597c1051-1bionic ceph-mgr=17.0.0-691-g597c1051-1bionic ceph-common=17.0.0-691-g597c1051-1bionic ceph-fuse=17.0.0-691-g597c1051-1bionic ceph-test=17.0.0-691-g597c1051-1bionic radosgw=17.0.0-691-g597c1051-1bionic python3-rados=17.0.0-691-g597c1051-1bionic python3-rgw=17.0.0-691-g597c1051-1bionic python3-cephfs=17.0.0-691-g597c1051-1bionic python3-rbd=17.0.0-691-g597c1051-1bionic libcephfs2=17.0.0-691-g597c1051-1bionic libcephfs-dev=17.0.0-691-g597c1051-1bionic librados2=17.0.0-691-g597c1051-1bionic librbd1=17.0.0-691-g597c1051-1bionic rbd-fuse=17.0.0-691-g597c1051-1bionic 2021-02-11T00:25:39.305 INFO:teuthology.orchestra.run.smithi095.stdout:Reading package lists... 2021-02-11T00:25:39.345 DEBUG:teuthology.orchestra.run.smithi095:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-691-g597c1051-1bionic cephadm=17.0.0-691-g597c1051-1bionic ceph-mds=17.0.0-691-g597c1051-1bionic ceph-mgr=17.0.0-691-g597c1051-1bionic ceph-common=17.0.0-691-g597c1051-1bionic ceph-fuse=17.0.0-691-g597c1051-1bionic ceph-test=17.0.0-691-g597c1051-1bionic radosgw=17.0.0-691-g597c1051-1bionic python3-rados=17.0.0-691-g597c1051-1bionic python3-rgw=17.0.0-691-g597c1051-1bionic python3-cephfs=17.0.0-691-g597c1051-1bionic python3-rbd=17.0.0-691-g597c1051-1bionic libcephfs2=17.0.0-691-g597c1051-1bionic libcephfs-dev=17.0.0-691-g597c1051-1bionic librados2=17.0.0-691-g597c1051-1bionic librbd1=17.0.0-691-g597c1051-1bionic rbd-fuse=17.0.0-691-g597c1051-1bionic 2021-02-11T00:25:39.369 INFO:teuthology.orchestra.run.smithi057.stdout:Reading package lists... 2021-02-11T00:25:39.439 INFO:teuthology.orchestra.run.smithi095.stdout:Reading package lists... 2021-02-11T00:25:39.634 INFO:teuthology.orchestra.run.smithi057.stdout:Building dependency tree... 2021-02-11T00:25:39.634 INFO:teuthology.orchestra.run.smithi057.stdout:Reading state information... 2021-02-11T00:25:39.689 INFO:teuthology.orchestra.run.smithi095.stdout:Building dependency tree... 2021-02-11T00:25:39.690 INFO:teuthology.orchestra.run.smithi095.stdout:Reading state information... 2021-02-11T00:25:39.813 INFO:teuthology.orchestra.run.smithi057.stdout:The following additional packages will be installed: 2021-02-11T00:25:39.814 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-02-11T00:25:39.815 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-02-11T00:25:39.815 INFO:teuthology.orchestra.run.smithi057.stdout: libjaeger libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-02-11T00:25:39.815 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-02-11T00:25:39.815 INFO:teuthology.orchestra.run.smithi057.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-02-11T00:25:39.815 INFO:teuthology.orchestra.run.smithi057.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-11T00:25:39.815 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-02-11T00:25:39.815 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-02-11T00:25:39.815 INFO:teuthology.orchestra.run.smithi057.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-02-11T00:25:39.815 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-02-11T00:25:39.815 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-02-11T00:25:39.816 INFO:teuthology.orchestra.run.smithi057.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-02-11T00:25:39.816 INFO:teuthology.orchestra.run.smithi057.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-02-11T00:25:39.816 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-02-11T00:25:39.816 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-02-11T00:25:39.816 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-02-11T00:25:39.816 INFO:teuthology.orchestra.run.smithi057.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-02-11T00:25:39.816 INFO:teuthology.orchestra.run.smithi057.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-02-11T00:25:39.816 INFO:teuthology.orchestra.run.smithi057.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-02-11T00:25:39.816 INFO:teuthology.orchestra.run.smithi057.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-02-11T00:25:39.816 INFO:teuthology.orchestra.run.smithi057.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-02-11T00:25:39.816 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-02-11T00:25:39.817 INFO:teuthology.orchestra.run.smithi057.stdout:Suggested packages: 2021-02-11T00:25:39.817 INFO:teuthology.orchestra.run.smithi057.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-02-11T00:25:39.817 INFO:teuthology.orchestra.run.smithi057.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-02-11T00:25:39.817 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-02-11T00:25:39.817 INFO:teuthology.orchestra.run.smithi057.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-02-11T00:25:39.817 INFO:teuthology.orchestra.run.smithi057.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-02-11T00:25:39.817 INFO:teuthology.orchestra.run.smithi057.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-02-11T00:25:39.818 INFO:teuthology.orchestra.run.smithi057.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-02-11T00:25:39.818 INFO:teuthology.orchestra.run.smithi057.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-02-11T00:25:39.818 INFO:teuthology.orchestra.run.smithi057.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-02-11T00:25:39.818 INFO:teuthology.orchestra.run.smithi057.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-02-11T00:25:39.818 INFO:teuthology.orchestra.run.smithi057.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-02-11T00:25:39.818 INFO:teuthology.orchestra.run.smithi057.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-02-11T00:25:39.818 INFO:teuthology.orchestra.run.smithi057.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-02-11T00:25:39.818 INFO:teuthology.orchestra.run.smithi057.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-02-11T00:25:39.819 INFO:teuthology.orchestra.run.smithi057.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-02-11T00:25:39.855 INFO:teuthology.orchestra.run.smithi095.stdout:The following additional packages will be installed: 2021-02-11T00:25:39.855 INFO:teuthology.orchestra.run.smithi095.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-02-11T00:25:39.856 INFO:teuthology.orchestra.run.smithi095.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-02-11T00:25:39.856 INFO:teuthology.orchestra.run.smithi095.stdout: libjaeger libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-02-11T00:25:39.856 INFO:teuthology.orchestra.run.smithi095.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-02-11T00:25:39.856 INFO:teuthology.orchestra.run.smithi095.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-02-11T00:25:39.856 INFO:teuthology.orchestra.run.smithi095.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-11T00:25:39.856 INFO:teuthology.orchestra.run.smithi095.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-02-11T00:25:39.856 INFO:teuthology.orchestra.run.smithi095.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-02-11T00:25:39.857 INFO:teuthology.orchestra.run.smithi095.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-02-11T00:25:39.857 INFO:teuthology.orchestra.run.smithi095.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-02-11T00:25:39.857 INFO:teuthology.orchestra.run.smithi095.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-02-11T00:25:39.857 INFO:teuthology.orchestra.run.smithi095.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-02-11T00:25:39.857 INFO:teuthology.orchestra.run.smithi095.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-02-11T00:25:39.857 INFO:teuthology.orchestra.run.smithi095.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-02-11T00:25:39.857 INFO:teuthology.orchestra.run.smithi095.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-02-11T00:25:39.857 INFO:teuthology.orchestra.run.smithi095.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-02-11T00:25:39.858 INFO:teuthology.orchestra.run.smithi095.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-02-11T00:25:39.858 INFO:teuthology.orchestra.run.smithi095.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-02-11T00:25:39.858 INFO:teuthology.orchestra.run.smithi095.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-02-11T00:25:39.858 INFO:teuthology.orchestra.run.smithi095.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-02-11T00:25:39.858 INFO:teuthology.orchestra.run.smithi095.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-02-11T00:25:39.858 INFO:teuthology.orchestra.run.smithi095.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-02-11T00:25:39.859 INFO:teuthology.orchestra.run.smithi095.stdout:Suggested packages: 2021-02-11T00:25:39.859 INFO:teuthology.orchestra.run.smithi095.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-02-11T00:25:39.859 INFO:teuthology.orchestra.run.smithi095.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-02-11T00:25:39.859 INFO:teuthology.orchestra.run.smithi095.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-02-11T00:25:39.859 INFO:teuthology.orchestra.run.smithi095.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-02-11T00:25:39.859 INFO:teuthology.orchestra.run.smithi095.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-02-11T00:25:39.860 INFO:teuthology.orchestra.run.smithi095.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-02-11T00:25:39.860 INFO:teuthology.orchestra.run.smithi095.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-02-11T00:25:39.860 INFO:teuthology.orchestra.run.smithi095.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-02-11T00:25:39.860 INFO:teuthology.orchestra.run.smithi095.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-02-11T00:25:39.860 INFO:teuthology.orchestra.run.smithi095.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-02-11T00:25:39.860 INFO:teuthology.orchestra.run.smithi095.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-02-11T00:25:39.860 INFO:teuthology.orchestra.run.smithi095.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-02-11T00:25:39.860 INFO:teuthology.orchestra.run.smithi095.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-02-11T00:25:39.860 INFO:teuthology.orchestra.run.smithi095.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-02-11T00:25:39.861 INFO:teuthology.orchestra.run.smithi095.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-02-11T00:25:39.926 INFO:teuthology.orchestra.run.smithi057.stdout:The following NEW packages will be installed: 2021-02-11T00:25:39.927 INFO:teuthology.orchestra.run.smithi057.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-02-11T00:25:39.927 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-02-11T00:25:39.927 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-02-11T00:25:39.927 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libleveldb1v5 liblttng-ust0 2021-02-11T00:25:39.928 INFO:teuthology.orchestra.run.smithi057.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-02-11T00:25:39.928 INFO:teuthology.orchestra.run.smithi057.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-02-11T00:25:39.928 INFO:teuthology.orchestra.run.smithi057.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-02-11T00:25:39.928 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-11T00:25:39.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-02-11T00:25:39.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-02-11T00:25:39.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-02-11T00:25:39.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-02-11T00:25:39.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-02-11T00:25:39.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-02-11T00:25:39.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-02-11T00:25:39.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-02-11T00:25:39.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-02-11T00:25:39.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-02-11T00:25:39.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-02-11T00:25:39.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-02-11T00:25:39.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-02-11T00:25:39.930 INFO:teuthology.orchestra.run.smithi057.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-02-11T00:25:39.930 INFO:teuthology.orchestra.run.smithi057.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-02-11T00:25:39.930 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-02-11T00:25:39.930 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-02-11T00:25:39.931 INFO:teuthology.orchestra.run.smithi057.stdout:The following packages will be upgraded: 2021-02-11T00:25:39.931 INFO:teuthology.orchestra.run.smithi057.stdout: librados2 librbd1 2021-02-11T00:25:39.958 INFO:teuthology.orchestra.run.smithi095.stdout:The following NEW packages will be installed: 2021-02-11T00:25:39.959 INFO:teuthology.orchestra.run.smithi095.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-02-11T00:25:39.959 INFO:teuthology.orchestra.run.smithi095.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-02-11T00:25:39.959 INFO:teuthology.orchestra.run.smithi095.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-02-11T00:25:39.960 INFO:teuthology.orchestra.run.smithi095.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libleveldb1v5 liblttng-ust0 2021-02-11T00:25:39.960 INFO:teuthology.orchestra.run.smithi095.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-02-11T00:25:39.960 INFO:teuthology.orchestra.run.smithi095.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-02-11T00:25:39.960 INFO:teuthology.orchestra.run.smithi095.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-02-11T00:25:39.960 INFO:teuthology.orchestra.run.smithi095.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-02-11T00:25:39.960 INFO:teuthology.orchestra.run.smithi095.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-02-11T00:25:39.961 INFO:teuthology.orchestra.run.smithi095.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-02-11T00:25:39.961 INFO:teuthology.orchestra.run.smithi095.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-02-11T00:25:39.961 INFO:teuthology.orchestra.run.smithi095.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-02-11T00:25:39.961 INFO:teuthology.orchestra.run.smithi095.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-02-11T00:25:39.961 INFO:teuthology.orchestra.run.smithi095.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-02-11T00:25:39.961 INFO:teuthology.orchestra.run.smithi095.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-02-11T00:25:39.961 INFO:teuthology.orchestra.run.smithi095.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-02-11T00:25:39.961 INFO:teuthology.orchestra.run.smithi095.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-02-11T00:25:39.961 INFO:teuthology.orchestra.run.smithi095.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-02-11T00:25:39.961 INFO:teuthology.orchestra.run.smithi095.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-02-11T00:25:39.961 INFO:teuthology.orchestra.run.smithi095.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-02-11T00:25:39.961 INFO:teuthology.orchestra.run.smithi095.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-02-11T00:25:39.961 INFO:teuthology.orchestra.run.smithi095.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-02-11T00:25:39.961 INFO:teuthology.orchestra.run.smithi095.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-02-11T00:25:39.962 INFO:teuthology.orchestra.run.smithi095.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-02-11T00:25:39.962 INFO:teuthology.orchestra.run.smithi095.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-02-11T00:25:39.962 INFO:teuthology.orchestra.run.smithi095.stdout:The following packages will be upgraded: 2021-02-11T00:25:39.962 INFO:teuthology.orchestra.run.smithi095.stdout: librados2 librbd1 2021-02-11T00:25:40.120 INFO:teuthology.orchestra.run.smithi057.stdout:2 upgraded, 111 newly installed, 0 to remove and 19 not upgraded. 2021-02-11T00:25:40.120 INFO:teuthology.orchestra.run.smithi057.stdout:Need to get 167 MB of archives. 2021-02-11T00:25:40.120 INFO:teuthology.orchestra.run.smithi057.stdout:After this operation, 778 MB of additional disk space will be used. 2021-02-11T00:25:40.120 INFO:teuthology.orchestra.run.smithi057.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-02-11T00:25:40.148 INFO:teuthology.orchestra.run.smithi095.stdout:2 upgraded, 111 newly installed, 0 to remove and 19 not upgraded. 2021-02-11T00:25:40.148 INFO:teuthology.orchestra.run.smithi095.stdout:Need to get 167 MB of archives. 2021-02-11T00:25:40.149 INFO:teuthology.orchestra.run.smithi095.stdout:After this operation, 778 MB of additional disk space will be used. 2021-02-11T00:25:40.149 INFO:teuthology.orchestra.run.smithi095.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-02-11T00:25:40.201 INFO:teuthology.orchestra.run.smithi057.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-02-11T00:25:40.227 INFO:teuthology.orchestra.run.smithi095.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-02-11T00:25:40.457 INFO:teuthology.orchestra.run.smithi057.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-02-11T00:25:40.467 INFO:teuthology.orchestra.run.smithi057.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-02-11T00:25:40.476 INFO:teuthology.orchestra.run.smithi095.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-02-11T00:25:40.487 INFO:teuthology.orchestra.run.smithi095.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-02-11T00:25:40.487 INFO:teuthology.orchestra.run.smithi057.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-02-11T00:25:40.496 INFO:teuthology.orchestra.run.smithi095.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-02-11T00:25:40.585 INFO:teuthology.orchestra.run.smithi057.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-02-11T00:25:40.588 INFO:teuthology.orchestra.run.smithi057.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-02-11T00:25:40.596 INFO:teuthology.orchestra.run.smithi095.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-02-11T00:25:40.598 INFO:teuthology.orchestra.run.smithi095.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-02-11T00:25:40.617 INFO:teuthology.orchestra.run.smithi057.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-02-11T00:25:40.624 INFO:teuthology.orchestra.run.smithi057.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-02-11T00:25:40.631 INFO:teuthology.orchestra.run.smithi095.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-02-11T00:25:40.631 INFO:teuthology.orchestra.run.smithi057.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-02-11T00:25:40.638 INFO:teuthology.orchestra.run.smithi095.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-02-11T00:25:40.642 INFO:teuthology.orchestra.run.smithi057.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-02-11T00:25:40.646 INFO:teuthology.orchestra.run.smithi095.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-02-11T00:25:40.650 INFO:teuthology.orchestra.run.smithi057.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-02-11T00:25:40.657 INFO:teuthology.orchestra.run.smithi095.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-02-11T00:25:40.665 INFO:teuthology.orchestra.run.smithi095.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-02-11T00:25:40.674 INFO:teuthology.orchestra.run.smithi057.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-02-11T00:25:40.681 INFO:teuthology.orchestra.run.smithi057.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-02-11T00:25:40.690 INFO:teuthology.orchestra.run.smithi095.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-02-11T00:25:40.694 INFO:teuthology.orchestra.run.smithi095.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-02-11T00:25:40.700 INFO:teuthology.orchestra.run.smithi057.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-02-11T00:25:40.711 INFO:teuthology.orchestra.run.smithi057.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-02-11T00:25:40.713 INFO:teuthology.orchestra.run.smithi057.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-02-11T00:25:40.713 INFO:teuthology.orchestra.run.smithi095.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-02-11T00:25:40.724 INFO:teuthology.orchestra.run.smithi095.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-02-11T00:25:40.725 INFO:teuthology.orchestra.run.smithi095.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-02-11T00:25:40.779 INFO:teuthology.orchestra.run.smithi057.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-02-11T00:25:40.780 INFO:teuthology.orchestra.run.smithi057.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-02-11T00:25:40.782 INFO:teuthology.orchestra.run.smithi057.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-02-11T00:25:40.783 INFO:teuthology.orchestra.run.smithi057.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-02-11T00:25:40.783 INFO:teuthology.orchestra.run.smithi057.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-02-11T00:25:40.786 INFO:teuthology.orchestra.run.smithi057.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-02-11T00:25:40.788 INFO:teuthology.orchestra.run.smithi057.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-02-11T00:25:40.792 INFO:teuthology.orchestra.run.smithi095.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-02-11T00:25:40.793 INFO:teuthology.orchestra.run.smithi095.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-02-11T00:25:40.795 INFO:teuthology.orchestra.run.smithi095.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-02-11T00:25:40.795 INFO:teuthology.orchestra.run.smithi095.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-02-11T00:25:40.796 INFO:teuthology.orchestra.run.smithi095.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-02-11T00:25:40.798 INFO:teuthology.orchestra.run.smithi095.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-02-11T00:25:40.802 INFO:teuthology.orchestra.run.smithi095.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-02-11T00:25:40.859 INFO:teuthology.orchestra.run.smithi057.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-02-11T00:25:40.860 INFO:teuthology.orchestra.run.smithi057.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-02-11T00:25:40.870 INFO:teuthology.orchestra.run.smithi095.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-02-11T00:25:40.871 INFO:teuthology.orchestra.run.smithi095.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-02-11T00:25:40.880 INFO:teuthology.orchestra.run.smithi057.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-02-11T00:25:40.890 INFO:teuthology.orchestra.run.smithi095.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-02-11T00:25:40.939 INFO:teuthology.orchestra.run.smithi057.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-02-11T00:25:40.940 INFO:teuthology.orchestra.run.smithi057.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-02-11T00:25:40.941 INFO:teuthology.orchestra.run.smithi057.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-02-11T00:25:40.944 INFO:teuthology.orchestra.run.smithi057.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-02-11T00:25:40.948 INFO:teuthology.orchestra.run.smithi095.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-02-11T00:25:40.949 INFO:teuthology.orchestra.run.smithi095.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-02-11T00:25:40.950 INFO:teuthology.orchestra.run.smithi095.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-02-11T00:25:40.952 INFO:teuthology.orchestra.run.smithi057.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-02-11T00:25:40.954 INFO:teuthology.orchestra.run.smithi095.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-02-11T00:25:40.956 INFO:teuthology.orchestra.run.smithi057.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-02-11T00:25:40.957 INFO:teuthology.orchestra.run.smithi057.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-02-11T00:25:40.962 INFO:teuthology.orchestra.run.smithi095.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-02-11T00:25:40.966 INFO:teuthology.orchestra.run.smithi095.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-02-11T00:25:40.966 INFO:teuthology.orchestra.run.smithi095.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-02-11T00:25:41.018 INFO:teuthology.orchestra.run.smithi057.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-02-11T00:25:41.027 INFO:teuthology.orchestra.run.smithi095.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-02-11T00:25:41.108 INFO:teuthology.orchestra.run.smithi057.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-02-11T00:25:41.108 INFO:teuthology.orchestra.run.smithi057.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-02-11T00:25:41.114 INFO:teuthology.orchestra.run.smithi095.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-02-11T00:25:41.115 INFO:teuthology.orchestra.run.smithi095.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-02-11T00:25:41.385 INFO:teuthology.orchestra.run.smithi057.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-02-11T00:25:41.385 INFO:teuthology.orchestra.run.smithi095.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-02-11T00:25:41.427 INFO:teuthology.orchestra.run.smithi095.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-02-11T00:25:41.429 INFO:teuthology.orchestra.run.smithi095.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-02-11T00:25:41.436 INFO:teuthology.orchestra.run.smithi057.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-02-11T00:25:41.437 INFO:teuthology.orchestra.run.smithi057.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-02-11T00:25:41.463 INFO:teuthology.orchestra.run.smithi095.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-02-11T00:25:41.463 INFO:teuthology.orchestra.run.smithi095.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-02-11T00:25:41.463 INFO:teuthology.orchestra.run.smithi057.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-02-11T00:25:41.463 INFO:teuthology.orchestra.run.smithi095.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2021-02-11T00:25:41.464 INFO:teuthology.orchestra.run.smithi057.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-02-11T00:25:41.464 INFO:teuthology.orchestra.run.smithi095.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-02-11T00:25:41.464 INFO:teuthology.orchestra.run.smithi057.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2021-02-11T00:25:41.465 INFO:teuthology.orchestra.run.smithi095.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-02-11T00:25:41.465 INFO:teuthology.orchestra.run.smithi057.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-02-11T00:25:41.465 INFO:teuthology.orchestra.run.smithi095.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-02-11T00:25:41.466 INFO:teuthology.orchestra.run.smithi057.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-02-11T00:25:41.466 INFO:teuthology.orchestra.run.smithi095.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-02-11T00:25:41.466 INFO:teuthology.orchestra.run.smithi057.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-02-11T00:25:41.467 INFO:teuthology.orchestra.run.smithi057.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-02-11T00:25:41.542 INFO:teuthology.orchestra.run.smithi095.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-02-11T00:25:41.544 INFO:teuthology.orchestra.run.smithi095.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-02-11T00:25:41.545 INFO:teuthology.orchestra.run.smithi057.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-02-11T00:25:41.547 INFO:teuthology.orchestra.run.smithi057.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-02-11T00:25:41.547 INFO:teuthology.orchestra.run.smithi095.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-02-11T00:25:41.548 INFO:teuthology.orchestra.run.smithi095.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-02-11T00:25:41.552 INFO:teuthology.orchestra.run.smithi095.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-02-11T00:25:41.554 INFO:teuthology.orchestra.run.smithi057.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-02-11T00:25:41.556 INFO:teuthology.orchestra.run.smithi057.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-02-11T00:25:41.558 INFO:teuthology.orchestra.run.smithi057.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-02-11T00:25:41.621 INFO:teuthology.orchestra.run.smithi095.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-02-11T00:25:41.621 INFO:teuthology.orchestra.run.smithi095.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-02-11T00:25:41.623 INFO:teuthology.orchestra.run.smithi095.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-02-11T00:25:41.624 INFO:teuthology.orchestra.run.smithi057.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-02-11T00:25:41.625 INFO:teuthology.orchestra.run.smithi095.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-02-11T00:25:41.625 INFO:teuthology.orchestra.run.smithi057.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-02-11T00:25:41.626 INFO:teuthology.orchestra.run.smithi095.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-02-11T00:25:41.630 INFO:teuthology.orchestra.run.smithi057.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-02-11T00:25:41.633 INFO:teuthology.orchestra.run.smithi057.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-02-11T00:25:41.635 INFO:teuthology.orchestra.run.smithi057.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-02-11T00:25:41.699 INFO:teuthology.orchestra.run.smithi095.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-02-11T00:25:41.702 INFO:teuthology.orchestra.run.smithi095.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-02-11T00:25:41.704 INFO:teuthology.orchestra.run.smithi057.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-02-11T00:25:41.708 INFO:teuthology.orchestra.run.smithi095.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-02-11T00:25:41.710 INFO:teuthology.orchestra.run.smithi057.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-02-11T00:25:41.712 INFO:teuthology.orchestra.run.smithi095.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-02-11T00:25:41.712 INFO:teuthology.orchestra.run.smithi095.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-02-11T00:25:41.718 INFO:teuthology.orchestra.run.smithi057.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-02-11T00:25:41.720 INFO:teuthology.orchestra.run.smithi057.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-02-11T00:25:41.721 INFO:teuthology.orchestra.run.smithi057.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-02-11T00:25:41.777 INFO:teuthology.orchestra.run.smithi095.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-02-11T00:25:41.778 INFO:teuthology.orchestra.run.smithi095.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-02-11T00:25:41.778 INFO:teuthology.orchestra.run.smithi095.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-02-11T00:25:41.781 INFO:teuthology.orchestra.run.smithi095.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-02-11T00:25:41.781 INFO:teuthology.orchestra.run.smithi095.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-02-11T00:25:41.783 INFO:teuthology.orchestra.run.smithi057.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-02-11T00:25:41.784 INFO:teuthology.orchestra.run.smithi057.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-02-11T00:25:41.786 INFO:teuthology.orchestra.run.smithi057.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-02-11T00:25:41.791 INFO:teuthology.orchestra.run.smithi057.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-02-11T00:25:41.792 INFO:teuthology.orchestra.run.smithi057.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-02-11T00:25:41.855 INFO:teuthology.orchestra.run.smithi095.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-02-11T00:25:41.856 INFO:teuthology.orchestra.run.smithi095.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-02-11T00:25:41.857 INFO:teuthology.orchestra.run.smithi095.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-02-11T00:25:41.863 INFO:teuthology.orchestra.run.smithi057.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-02-11T00:25:41.864 INFO:teuthology.orchestra.run.smithi057.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-02-11T00:25:41.867 INFO:teuthology.orchestra.run.smithi057.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-02-11T00:25:41.879 INFO:teuthology.orchestra.run.smithi095.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-02-11T00:25:41.880 INFO:teuthology.orchestra.run.smithi095.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-02-11T00:25:41.887 INFO:teuthology.orchestra.run.smithi057.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-02-11T00:25:41.887 INFO:teuthology.orchestra.run.smithi057.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-02-11T00:25:41.933 INFO:teuthology.orchestra.run.smithi095.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-02-11T00:25:41.942 INFO:teuthology.orchestra.run.smithi057.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-02-11T00:25:42.029 INFO:teuthology.orchestra.run.smithi095.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-02-11T00:25:42.029 INFO:teuthology.orchestra.run.smithi095.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-02-11T00:25:42.030 INFO:teuthology.orchestra.run.smithi095.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-02-11T00:25:42.034 INFO:teuthology.orchestra.run.smithi095.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-02-11T00:25:42.034 INFO:teuthology.orchestra.run.smithi095.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-02-11T00:25:42.035 INFO:teuthology.orchestra.run.smithi095.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-02-11T00:25:42.035 INFO:teuthology.orchestra.run.smithi095.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-02-11T00:25:42.037 INFO:teuthology.orchestra.run.smithi095.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-02-11T00:25:42.037 INFO:teuthology.orchestra.run.smithi095.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-02-11T00:25:42.044 INFO:teuthology.orchestra.run.smithi057.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-02-11T00:25:42.044 INFO:teuthology.orchestra.run.smithi057.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-02-11T00:25:42.045 INFO:teuthology.orchestra.run.smithi057.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-02-11T00:25:42.050 INFO:teuthology.orchestra.run.smithi057.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-02-11T00:25:42.051 INFO:teuthology.orchestra.run.smithi057.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-02-11T00:25:42.051 INFO:teuthology.orchestra.run.smithi057.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-02-11T00:25:42.102 INFO:teuthology.orchestra.run.smithi057.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-02-11T00:25:42.105 INFO:teuthology.orchestra.run.smithi057.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-02-11T00:25:42.106 INFO:teuthology.orchestra.run.smithi057.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-02-11T00:25:45.715 INFO:teuthology.orchestra.run.smithi057.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 libjaeger amd64 17.0.0-691-g597c1051-1bionic [3,720 B] 2021-02-11T00:25:45.738 INFO:teuthology.orchestra.run.smithi095.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 libjaeger amd64 17.0.0-691-g597c1051-1bionic [3,720 B] 2021-02-11T00:25:45.763 INFO:teuthology.orchestra.run.smithi057.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 17.0.0-691-g597c1051-1bionic [3,124 kB] 2021-02-11T00:25:45.786 INFO:teuthology.orchestra.run.smithi095.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 17.0.0-691-g597c1051-1bionic [3,124 kB] 2021-02-11T00:25:46.163 INFO:teuthology.orchestra.run.smithi057.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 17.0.0-691-g597c1051-1bionic [3,163 kB] 2021-02-11T00:25:46.270 INFO:teuthology.orchestra.run.smithi095.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 17.0.0-691-g597c1051-1bionic [3,163 kB] 2021-02-11T00:25:53.166 INFO:teuthology.orchestra.run.smithi057.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 17.0.0-691-g597c1051-1bionic [664 kB] 2021-02-11T00:25:53.263 INFO:teuthology.orchestra.run.smithi057.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 17.0.0-691-g597c1051-1bionic [339 kB] 2021-02-11T00:25:53.303 INFO:teuthology.orchestra.run.smithi057.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 17.0.0-691-g597c1051-1bionic [21.7 kB] 2021-02-11T00:25:53.304 INFO:teuthology.orchestra.run.smithi057.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 17.0.0-691-g597c1051-1bionic [163 kB] 2021-02-11T00:25:53.309 INFO:teuthology.orchestra.run.smithi057.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 17.0.0-691-g597c1051-1bionic [29.2 kB] 2021-02-11T00:25:53.310 INFO:teuthology.orchestra.run.smithi057.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 17.0.0-691-g597c1051-1bionic [335 kB] 2021-02-11T00:25:53.365 INFO:teuthology.orchestra.run.smithi057.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 17.0.0-691-g597c1051-1bionic [3,350 kB] 2021-02-11T00:25:53.472 INFO:teuthology.orchestra.run.smithi095.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 17.0.0-691-g597c1051-1bionic [664 kB] 2021-02-11T00:25:53.547 INFO:teuthology.orchestra.run.smithi095.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 17.0.0-691-g597c1051-1bionic [339 kB] 2021-02-11T00:25:53.610 INFO:teuthology.orchestra.run.smithi095.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 17.0.0-691-g597c1051-1bionic [21.7 kB] 2021-02-11T00:25:53.611 INFO:teuthology.orchestra.run.smithi095.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 17.0.0-691-g597c1051-1bionic [163 kB] 2021-02-11T00:25:53.616 INFO:teuthology.orchestra.run.smithi095.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 17.0.0-691-g597c1051-1bionic [29.2 kB] 2021-02-11T00:25:53.617 INFO:teuthology.orchestra.run.smithi095.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 17.0.0-691-g597c1051-1bionic [335 kB] 2021-02-11T00:25:53.679 INFO:teuthology.orchestra.run.smithi095.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 17.0.0-691-g597c1051-1bionic [3,350 kB] 2021-02-11T00:25:53.754 INFO:teuthology.orchestra.run.smithi057.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 17.0.0-691-g597c1051-1bionic [99.3 kB] 2021-02-11T00:25:53.757 INFO:teuthology.orchestra.run.smithi057.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 17.0.0-691-g597c1051-1bionic [414 kB] 2021-02-11T00:25:53.817 INFO:teuthology.orchestra.run.smithi057.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 17.0.0-691-g597c1051-1bionic [21.2 MB] 2021-02-11T00:25:54.220 INFO:teuthology.orchestra.run.smithi095.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 17.0.0-691-g597c1051-1bionic [99.3 kB] 2021-02-11T00:25:54.224 INFO:teuthology.orchestra.run.smithi095.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 17.0.0-691-g597c1051-1bionic [414 kB] 2021-02-11T00:25:54.293 INFO:teuthology.orchestra.run.smithi095.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 17.0.0-691-g597c1051-1bionic [21.2 MB] 2021-02-11T00:25:56.237 INFO:teuthology.orchestra.run.smithi057.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 17.0.0-691-g597c1051-1bionic [5,626 kB] 2021-02-11T00:25:56.845 INFO:teuthology.orchestra.run.smithi057.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 17.0.0-691-g597c1051-1bionic [185 kB] 2021-02-11T00:25:56.858 INFO:teuthology.orchestra.run.smithi057.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 17.0.0-691-g597c1051-1bionic [1,365 kB] 2021-02-11T00:26:00.138 INFO:teuthology.orchestra.run.smithi057.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 17.0.0-691-g597c1051-1bionic [6,659 kB] 2021-02-11T00:26:00.755 INFO:teuthology.orchestra.run.smithi095.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 17.0.0-691-g597c1051-1bionic [5,626 kB] 2021-02-11T00:26:00.842 INFO:teuthology.orchestra.run.smithi057.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 17.0.0-691-g597c1051-1bionic [24.9 MB] 2021-02-11T00:26:01.584 INFO:teuthology.orchestra.run.smithi095.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 17.0.0-691-g597c1051-1bionic [185 kB] 2021-02-11T00:26:01.590 INFO:teuthology.orchestra.run.smithi095.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 17.0.0-691-g597c1051-1bionic [1,365 kB] 2021-02-11T00:26:02.669 INFO:teuthology.orchestra.run.smithi095.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 17.0.0-691-g597c1051-1bionic [6,659 kB] 2021-02-11T00:26:03.937 INFO:teuthology.orchestra.run.smithi057.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 17.0.0-691-g597c1051-1bionic [3,812 B] 2021-02-11T00:26:03.938 INFO:teuthology.orchestra.run.smithi057.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 17.0.0-691-g597c1051-1bionic [771 kB] 2021-02-11T00:26:03.961 INFO:teuthology.orchestra.run.smithi095.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 17.0.0-691-g597c1051-1bionic [24.9 MB] 2021-02-11T00:26:04.013 INFO:teuthology.orchestra.run.smithi057.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 17.0.0-691-g597c1051-1bionic [2,153 kB] 2021-02-11T00:26:05.167 INFO:teuthology.orchestra.run.smithi057.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 17.0.0-691-g597c1051-1bionic [62.1 kB] 2021-02-11T00:26:05.978 INFO:teuthology.orchestra.run.smithi057.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 17.0.0-691-g597c1051-1bionic [67.2 kB] 2021-02-11T00:26:05.978 INFO:teuthology.orchestra.run.smithi057.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 17.0.0-691-g597c1051-1bionic [2,327 kB] 2021-02-11T00:26:05.979 INFO:teuthology.orchestra.run.smithi057.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 17.0.0-691-g597c1051-1bionic [7,886 kB] 2021-02-11T00:26:06.183 INFO:teuthology.orchestra.run.smithi057.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 17.0.0-691-g597c1051-1bionic [3,924 B] 2021-02-11T00:26:06.183 INFO:teuthology.orchestra.run.smithi057.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 17.0.0-691-g597c1051-1bionic [23.3 kB] 2021-02-11T00:26:06.183 INFO:teuthology.orchestra.run.smithi057.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 17.0.0-691-g597c1051-1bionic [45.3 MB] 2021-02-11T00:26:08.914 INFO:teuthology.orchestra.run.smithi095.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 17.0.0-691-g597c1051-1bionic [3,812 B] 2021-02-11T00:26:08.915 INFO:teuthology.orchestra.run.smithi095.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 17.0.0-691-g597c1051-1bionic [771 kB] 2021-02-11T00:26:09.059 INFO:teuthology.orchestra.run.smithi095.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 17.0.0-691-g597c1051-1bionic [2,153 kB] 2021-02-11T00:26:09.377 INFO:teuthology.orchestra.run.smithi095.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 17.0.0-691-g597c1051-1bionic [62.1 kB] 2021-02-11T00:26:09.382 INFO:teuthology.orchestra.run.smithi095.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 17.0.0-691-g597c1051-1bionic [67.2 kB] 2021-02-11T00:26:09.386 INFO:teuthology.orchestra.run.smithi095.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 17.0.0-691-g597c1051-1bionic [2,327 kB] 2021-02-11T00:26:09.681 INFO:teuthology.orchestra.run.smithi095.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 17.0.0-691-g597c1051-1bionic [7,886 kB] 2021-02-11T00:26:17.757 INFO:teuthology.orchestra.run.smithi095.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 17.0.0-691-g597c1051-1bionic [3,924 B] 2021-02-11T00:26:17.758 INFO:teuthology.orchestra.run.smithi095.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 17.0.0-691-g597c1051-1bionic [23.3 kB] 2021-02-11T00:26:17.758 INFO:teuthology.orchestra.run.smithi095.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 17.0.0-691-g597c1051-1bionic [45.3 MB] 2021-02-11T00:26:18.838 INFO:teuthology.orchestra.run.smithi057.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 17.0.0-691-g597c1051-1bionic [19.8 kB] 2021-02-11T00:26:18.874 INFO:teuthology.orchestra.run.smithi057.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 17.0.0-691-g597c1051-1bionic [10.3 MB] 2021-02-11T00:26:20.701 INFO:teuthology.orchestra.run.smithi057.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 17.0.0-691-g597c1051-1bionic [77.8 kB] 2021-02-11T00:26:21.450 INFO:teuthology.orchestra.run.smithi057.stdout:Fetched 167 MB in 41s (4,090 kB/s) 2021-02-11T00:26:21.562 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-uritemplate. 2021-02-11T00:26:21.614 INFO:teuthology.orchestra.run.smithi057.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137611 files and directories currently installed.) 2021-02-11T00:26:21.618 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-02-11T00:26:21.652 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-02-11T00:26:21.854 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libjaeger. 2021-02-11T00:26:21.868 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../001-libjaeger_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:21.904 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libjaeger (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:22.241 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-02-11T00:26:22.255 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../002-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-02-11T00:26:22.341 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-11T00:26:22.676 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../003-librbd1_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:22.779 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking librbd1 (17.0.0-691-g597c1051-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-11T00:26:23.429 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../004-librados2_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:23.550 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking librados2 (17.0.0-691-g597c1051-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-11T00:26:23.725 INFO:teuthology.orchestra.run.smithi095.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 17.0.0-691-g597c1051-1bionic [19.8 kB] 2021-02-11T00:26:23.725 INFO:teuthology.orchestra.run.smithi095.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 17.0.0-691-g597c1051-1bionic [10.3 MB] 2021-02-11T00:26:24.154 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libcephfs2. 2021-02-11T00:26:24.170 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../005-libcephfs2_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:24.204 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libcephfs2 (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:24.498 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-rados. 2021-02-11T00:26:24.514 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../006-python3-rados_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:24.556 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-rados (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:24.910 INFO:teuthology.orchestra.run.smithi095.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-datalog-genbacking/597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 17.0.0-691-g597c1051-1bionic [77.8 kB] 2021-02-11T00:26:24.910 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-02-11T00:26:24.924 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../007-python3-ceph-argparse_17.0.0-691-g597c1051-1bionic_all.deb ... 2021-02-11T00:26:24.968 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-ceph-argparse (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:25.187 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-cephfs. 2021-02-11T00:26:25.202 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../008-python3-cephfs_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:25.237 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-cephfs (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:25.498 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-ceph-common. 2021-02-11T00:26:25.512 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../009-python3-ceph-common_17.0.0-691-g597c1051-1bionic_all.deb ... 2021-02-11T00:26:25.548 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-ceph-common (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:25.623 INFO:teuthology.orchestra.run.smithi095.stdout:Fetched 167 MB in 45s (3,712 kB/s) 2021-02-11T00:26:25.738 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-uritemplate. 2021-02-11T00:26:25.784 INFO:teuthology.orchestra.run.smithi095.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137611 files and directories currently installed.) 2021-02-11T00:26:25.788 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-02-11T00:26:25.828 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-02-11T00:26:25.860 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-prettytable. 2021-02-11T00:26:25.875 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../010-python3-prettytable_0.7.2-3_all.deb ... 2021-02-11T00:26:25.918 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-02-11T00:26:26.147 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package libjaeger. 2021-02-11T00:26:26.163 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../001-libjaeger_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:26.187 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-rbd. 2021-02-11T00:26:26.197 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking libjaeger (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:26.201 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../011-python3-rbd_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:26.238 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-rbd (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:26.534 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-02-11T00:26:26.541 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-02-11T00:26:26.547 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../002-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-02-11T00:26:26.554 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../012-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-02-11T00:26:26.582 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-11T00:26:26.625 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-11T00:26:26.894 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-02-11T00:26:26.908 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../013-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-02-11T00:26:26.943 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../003-librbd1_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:26.952 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-11T00:26:27.046 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking librbd1 (17.0.0-691-g597c1051-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-11T00:26:27.288 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package librgw2. 2021-02-11T00:26:27.303 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../014-librgw2_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:27.346 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking librgw2 (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:27.706 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../004-librados2_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:27.834 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking librados2 (17.0.0-691-g597c1051-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-02-11T00:26:27.908 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-rgw. 2021-02-11T00:26:27.923 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../015-python3-rgw_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:27.958 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-rgw (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:28.252 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-02-11T00:26:28.267 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-02-11T00:26:28.311 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-11T00:26:28.404 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package libcephfs2. 2021-02-11T00:26:28.419 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../005-libcephfs2_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:28.454 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking libcephfs2 (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:28.622 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-02-11T00:26:28.636 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2021-02-11T00:26:28.680 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-02-11T00:26:28.748 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-rados. 2021-02-11T00:26:28.762 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../006-python3-rados_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:28.798 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-rados (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:28.992 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package liboath0. 2021-02-11T00:26:29.007 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2021-02-11T00:26:29.042 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-02-11T00:26:29.117 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-02-11T00:26:29.132 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../007-python3-ceph-argparse_17.0.0-691-g597c1051-1bionic_all.deb ... 2021-02-11T00:26:29.175 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-ceph-argparse (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:29.336 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libradosstriper1. 2021-02-11T00:26:29.351 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../019-libradosstriper1_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:29.395 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libradosstriper1 (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:29.419 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-cephfs. 2021-02-11T00:26:29.434 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../008-python3-cephfs_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:29.478 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-cephfs (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:29.756 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-ceph-common. 2021-02-11T00:26:29.771 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../009-python3-ceph-common_17.0.0-691-g597c1051-1bionic_all.deb ... 2021-02-11T00:26:29.805 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-ceph-common (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:29.831 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-common. 2021-02-11T00:26:29.845 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../020-ceph-common_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:29.898 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-common (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:30.067 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-prettytable. 2021-02-11T00:26:30.081 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../010-python3-prettytable_0.7.2-3_all.deb ... 2021-02-11T00:26:30.150 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-02-11T00:26:30.403 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-rbd. 2021-02-11T00:26:30.417 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../011-python3-rbd_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:30.461 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-rbd (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:30.830 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-02-11T00:26:30.845 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../012-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-02-11T00:26:30.880 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-11T00:26:31.167 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-02-11T00:26:31.182 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../013-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-02-11T00:26:31.217 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-11T00:26:31.536 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package librgw2. 2021-02-11T00:26:31.550 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../014-librgw2_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:31.594 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking librgw2 (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:31.937 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-base. 2021-02-11T00:26:31.950 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../021-ceph-base_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:31.996 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-base (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:32.163 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-rgw. 2021-02-11T00:26:32.178 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../015-python3-rgw_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:32.222 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-rgw (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:32.499 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-02-11T00:26:32.515 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-02-11T00:26:32.549 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-11T00:26:32.794 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-02-11T00:26:32.809 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2021-02-11T00:26:32.852 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-02-11T00:26:32.886 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-dateutil. 2021-02-11T00:26:32.902 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../022-python3-dateutil_2.6.1-1_all.deb ... 2021-02-11T00:26:32.936 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-02-11T00:26:33.222 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package liboath0. 2021-02-11T00:26:33.237 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2021-02-11T00:26:33.280 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-02-11T00:26:33.364 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-asn1crypto. 2021-02-11T00:26:33.377 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../023-python3-asn1crypto_0.24.0-1_all.deb ... 2021-02-11T00:26:33.422 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-02-11T00:26:33.574 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package libradosstriper1. 2021-02-11T00:26:33.590 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../019-libradosstriper1_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:33.624 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking libradosstriper1 (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:33.691 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-cffi-backend. 2021-02-11T00:26:33.705 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../024-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-02-11T00:26:33.749 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-02-11T00:26:33.961 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package ceph-common. 2021-02-11T00:26:33.974 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../020-ceph-common_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:34.011 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking ceph-common (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:34.068 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-cryptography. 2021-02-11T00:26:34.083 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../025-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-02-11T00:26:34.127 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-11T00:26:34.530 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-openssl. 2021-02-11T00:26:34.546 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../026-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-02-11T00:26:34.597 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-02-11T00:26:34.874 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-02-11T00:26:34.889 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.0.0-691-g597c1051-1bionic_all.deb ... 2021-02-11T00:26:34.933 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-mgr-modules-core (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:35.287 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-bcrypt. 2021-02-11T00:26:35.301 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-02-11T00:26:35.345 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-02-11T00:26:35.665 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-cherrypy3. 2021-02-11T00:26:35.679 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-2_all.deb ... 2021-02-11T00:26:35.723 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-02-11T00:26:35.899 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package ceph-base. 2021-02-11T00:26:35.912 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../021-ceph-base_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:35.958 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking ceph-base (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:36.110 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-jwt. 2021-02-11T00:26:36.124 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../030-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-02-11T00:26:36.168 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-02-11T00:26:36.479 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-logutils. 2021-02-11T00:26:36.493 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../031-python3-logutils_0.3.3-5_all.deb ... 2021-02-11T00:26:36.537 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-02-11T00:26:36.832 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-markupsafe. 2021-02-11T00:26:36.845 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../032-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-02-11T00:26:36.862 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-dateutil. 2021-02-11T00:26:36.877 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../022-python3-dateutil_2.6.1-1_all.deb ... 2021-02-11T00:26:36.882 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-02-11T00:26:36.921 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-02-11T00:26:37.193 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-mako. 2021-02-11T00:26:37.208 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../033-python3-mako_1.0.7+ds1-1_all.deb ... 2021-02-11T00:26:37.215 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-asn1crypto. 2021-02-11T00:26:37.231 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../023-python3-asn1crypto_0.24.0-1_all.deb ... 2021-02-11T00:26:37.251 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-02-11T00:26:37.274 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-02-11T00:26:37.535 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-cffi-backend. 2021-02-11T00:26:37.551 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../024-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-02-11T00:26:37.593 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-02-11T00:26:37.604 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-simplegeneric. 2021-02-11T00:26:37.619 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../034-python3-simplegeneric_0.8.1-1_all.deb ... 2021-02-11T00:26:37.671 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-02-11T00:26:37.870 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-cryptography. 2021-02-11T00:26:37.885 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../025-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-02-11T00:26:37.920 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-11T00:26:37.981 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-singledispatch. 2021-02-11T00:26:37.996 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../035-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-02-11T00:26:38.039 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-02-11T00:26:38.215 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-openssl. 2021-02-11T00:26:38.228 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../026-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-02-11T00:26:38.257 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-02-11T00:26:38.326 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-webob. 2021-02-11T00:26:38.342 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../036-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-02-11T00:26:38.385 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-02-11T00:26:38.510 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-02-11T00:26:38.523 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.0.0-691-g597c1051-1bionic_all.deb ... 2021-02-11T00:26:38.585 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking ceph-mgr-modules-core (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:38.705 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-bs4. 2021-02-11T00:26:38.718 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../037-python3-bs4_4.6.0-1_all.deb ... 2021-02-11T00:26:38.754 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-02-11T00:26:38.931 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-bcrypt. 2021-02-11T00:26:38.946 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-02-11T00:26:38.989 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-02-11T00:26:39.099 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-waitress. 2021-02-11T00:26:39.113 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../038-python3-waitress_1.0.1-1_all.deb ... 2021-02-11T00:26:39.166 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-02-11T00:26:39.267 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-cherrypy3. 2021-02-11T00:26:39.283 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-2_all.deb ... 2021-02-11T00:26:39.317 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-02-11T00:26:39.535 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-tempita. 2021-02-11T00:26:39.552 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../039-python3-tempita_0.5.2-2_all.deb ... 2021-02-11T00:26:39.602 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-02-11T00:26:39.678 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-jwt. 2021-02-11T00:26:39.694 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../030-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-02-11T00:26:39.745 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-02-11T00:26:39.905 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-paste. 2021-02-11T00:26:39.919 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../040-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-02-11T00:26:39.963 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-11T00:26:40.031 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-logutils. 2021-02-11T00:26:40.047 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../031-python3-logutils_0.3.3-5_all.deb ... 2021-02-11T00:26:40.090 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-02-11T00:26:40.291 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-02-11T00:26:40.305 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../041-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-02-11T00:26:40.357 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-02-11T00:26:40.393 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-markupsafe. 2021-02-11T00:26:40.406 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../032-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-02-11T00:26:40.451 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-02-11T00:26:40.660 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pastedeploy. 2021-02-11T00:26:40.674 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../042-python3-pastedeploy_1.5.2-4_all.deb ... 2021-02-11T00:26:40.710 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-02-11T00:26:40.745 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-mako. 2021-02-11T00:26:40.759 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../033-python3-mako_1.0.7+ds1-1_all.deb ... 2021-02-11T00:26:40.804 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-02-11T00:26:41.013 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-webtest. 2021-02-11T00:26:41.028 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../043-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-02-11T00:26:41.071 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-02-11T00:26:41.115 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-simplegeneric. 2021-02-11T00:26:41.130 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../034-python3-simplegeneric_0.8.1-1_all.deb ... 2021-02-11T00:26:41.173 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-02-11T00:26:41.390 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pecan. 2021-02-11T00:26:41.406 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../044-python3-pecan_1.2.1-2_all.deb ... 2021-02-11T00:26:41.449 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-02-11T00:26:41.518 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-singledispatch. 2021-02-11T00:26:41.531 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../035-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-02-11T00:26:41.577 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-02-11T00:26:41.829 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-werkzeug. 2021-02-11T00:26:41.844 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../045-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-02-11T00:26:41.887 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-11T00:26:41.888 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-webob. 2021-02-11T00:26:41.902 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../036-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-02-11T00:26:41.946 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-02-11T00:26:42.248 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-mgr. 2021-02-11T00:26:42.259 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-bs4. 2021-02-11T00:26:42.263 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../046-ceph-mgr_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:42.274 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../037-python3-bs4_4.6.0-1_all.deb ... 2021-02-11T00:26:42.306 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-mgr (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:42.317 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-02-11T00:26:42.654 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-waitress. 2021-02-11T00:26:42.667 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../038-python3-waitress_1.0.1-1_all.deb ... 2021-02-11T00:26:42.704 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-02-11T00:26:42.793 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-mon. 2021-02-11T00:26:42.808 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../047-ceph-mon_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:42.851 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-mon (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:43.024 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-tempita. 2021-02-11T00:26:43.038 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../039-python3-tempita_0.5.2-2_all.deb ... 2021-02-11T00:26:43.082 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-02-11T00:26:43.385 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-paste. 2021-02-11T00:26:43.399 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../040-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-02-11T00:26:43.443 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-11T00:26:43.772 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-02-11T00:26:43.788 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../041-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-02-11T00:26:43.823 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-osd. 2021-02-11T00:26:43.830 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-02-11T00:26:43.836 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../048-ceph-osd_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:43.881 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-osd (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:44.107 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-pastedeploy. 2021-02-11T00:26:44.124 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../042-python3-pastedeploy_1.5.2-4_all.deb ... 2021-02-11T00:26:44.166 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-02-11T00:26:44.544 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-webtest. 2021-02-11T00:26:44.558 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../043-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-02-11T00:26:44.611 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-02-11T00:26:44.947 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-pecan. 2021-02-11T00:26:44.962 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../044-python3-pecan_1.2.1-2_all.deb ... 2021-02-11T00:26:45.005 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-02-11T00:26:45.385 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-werkzeug. 2021-02-11T00:26:45.400 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../045-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-02-11T00:26:45.443 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-11T00:26:45.838 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package ceph-mgr. 2021-02-11T00:26:45.852 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../046-ceph-mgr_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:45.896 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking ceph-mgr (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:46.037 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph. 2021-02-11T00:26:46.052 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../049-ceph_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:46.095 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:46.374 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package ceph-mon. 2021-02-11T00:26:46.389 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../047-ceph-mon_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:46.414 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-fuse. 2021-02-11T00:26:46.430 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../050-ceph-fuse_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:46.433 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking ceph-mon (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:46.465 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-fuse (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:46.860 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-mds. 2021-02-11T00:26:46.877 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../051-ceph-mds_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:46.918 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-mds (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:47.412 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package ceph-osd. 2021-02-11T00:26:47.426 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../048-ceph-osd_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:47.429 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package cephadm. 2021-02-11T00:26:47.444 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../052-cephadm_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:47.470 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking ceph-osd (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:47.487 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking cephadm (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:47.807 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-jinja2. 2021-02-11T00:26:47.820 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../053-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-02-11T00:26:47.865 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-11T00:26:48.154 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-02-11T00:26:48.170 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../054-ceph-mgr-cephadm_17.0.0-691-g597c1051-1bionic_all.deb ... 2021-02-11T00:26:48.212 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-mgr-cephadm (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:48.532 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-repoze.lru. 2021-02-11T00:26:48.545 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../055-python3-repoze.lru_0.7-1_all.deb ... 2021-02-11T00:26:48.582 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-02-11T00:26:48.878 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-routes. 2021-02-11T00:26:48.894 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../056-python3-routes_2.4.1-1_all.deb ... 2021-02-11T00:26:48.936 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-02-11T00:26:49.263 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-02-11T00:26:49.278 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_17.0.0-691-g597c1051-1bionic_all.deb ... 2021-02-11T00:26:49.322 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-mgr-dashboard (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:49.510 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package ceph. 2021-02-11T00:26:49.525 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../049-ceph_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:49.577 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking ceph (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:49.913 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package ceph-fuse. 2021-02-11T00:26:49.927 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../050-ceph-fuse_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:49.971 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking ceph-fuse (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:50.193 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-numpy. 2021-02-11T00:26:50.210 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-02-11T00:26:50.251 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-11T00:26:50.392 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package ceph-mds. 2021-02-11T00:26:50.405 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../051-ceph-mds_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:50.442 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking ceph-mds (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:50.840 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-decorator. 2021-02-11T00:26:50.856 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-02-11T00:26:50.898 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-02-11T00:26:50.928 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package cephadm. 2021-02-11T00:26:50.944 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../052-cephadm_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:26:50.987 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking cephadm (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:51.209 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-scipy. 2021-02-11T00:26:51.224 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-02-11T00:26:51.267 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-02-11T00:26:51.331 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-jinja2. 2021-02-11T00:26:51.346 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../053-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-02-11T00:26:51.397 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-11T00:26:51.817 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-02-11T00:26:51.834 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../054-ceph-mgr-cephadm_17.0.0-691-g597c1051-1bionic_all.deb ... 2021-02-11T00:26:51.884 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking ceph-mgr-cephadm (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:52.215 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-repoze.lru. 2021-02-11T00:26:52.229 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../055-python3-repoze.lru_0.7-1_all.deb ... 2021-02-11T00:26:52.273 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-02-11T00:26:52.559 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-routes. 2021-02-11T00:26:52.575 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../056-python3-routes_2.4.1-1_all.deb ... 2021-02-11T00:26:52.617 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-02-11T00:26:52.688 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-02-11T00:26:52.706 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-02-11T00:26:52.746 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-02-11T00:26:52.963 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-02-11T00:26:52.978 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_17.0.0-691-g597c1051-1bionic_all.deb ... 2021-02-11T00:26:53.021 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking ceph-mgr-dashboard (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:53.219 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-joblib. 2021-02-11T00:26:53.233 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-02-11T00:26:53.278 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-joblib (0.11-1) ... 2021-02-11T00:26:53.608 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-sklearn. 2021-02-11T00:26:53.625 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-02-11T00:26:53.666 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-02-11T00:26:53.894 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-numpy. 2021-02-11T00:26:53.907 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-02-11T00:26:53.952 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-11T00:26:54.250 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-02-11T00:26:54.265 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_17.0.0-691-g597c1051-1bionic_all.deb ... 2021-02-11T00:26:54.308 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:54.591 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-decorator. 2021-02-11T00:26:54.606 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-02-11T00:26:54.649 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-02-11T00:26:54.969 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-scipy. 2021-02-11T00:26:54.983 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-02-11T00:26:55.027 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-02-11T00:26:55.507 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-monotonic. 2021-02-11T00:26:55.524 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-02-11T00:26:55.565 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-02-11T00:26:55.868 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-fasteners. 2021-02-11T00:26:55.884 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-02-11T00:26:55.960 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-02-11T00:26:56.261 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-02-11T00:26:56.271 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-httplib2. 2021-02-11T00:26:56.277 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-02-11T00:26:56.285 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2021-02-11T00:26:56.319 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-02-11T00:26:56.329 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-11T00:26:56.649 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pyasn1. 2021-02-11T00:26:56.662 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-02-11T00:26:56.707 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-02-11T00:26:57.044 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-02-11T00:26:57.057 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-02-11T00:26:57.102 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-11T00:26:57.192 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-joblib. 2021-02-11T00:26:57.206 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-02-11T00:26:57.250 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-joblib (0.11-1) ... 2021-02-11T00:26:57.404 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-rsa. 2021-02-11T00:26:57.421 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-02-11T00:26:57.463 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-02-11T00:26:57.599 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-sklearn. 2021-02-11T00:26:57.614 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-02-11T00:26:57.657 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-02-11T00:26:57.774 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-02-11T00:26:57.790 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-02-11T00:26:57.824 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-11T00:26:58.207 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-02-11T00:26:58.222 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_17.0.0-691-g597c1051-1bionic_all.deb ... 2021-02-11T00:26:58.261 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-oauth2client. 2021-02-11T00:26:58.276 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-02-11T00:26:58.281 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:26:58.319 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-02-11T00:26:58.656 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-setuptools. 2021-02-11T00:26:58.672 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-02-11T00:26:58.714 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-02-11T00:26:59.110 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-websocket. 2021-02-11T00:26:59.124 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-02-11T00:26:59.167 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-02-11T00:26:59.380 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-monotonic. 2021-02-11T00:26:59.394 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-02-11T00:26:59.438 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-02-11T00:26:59.504 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-kubernetes. 2021-02-11T00:26:59.518 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-02-11T00:26:59.562 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-11T00:26:59.753 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-fasteners. 2021-02-11T00:26:59.767 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-02-11T00:26:59.811 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-02-11T00:26:59.958 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-02-11T00:26:59.971 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_17.0.0-691-g597c1051-1bionic_all.deb ... 2021-02-11T00:27:00.016 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:00.122 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-httplib2. 2021-02-11T00:27:00.137 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2021-02-11T00:27:00.181 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-11T00:27:00.385 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-json-pointer. 2021-02-11T00:27:00.400 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-02-11T00:27:00.443 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-02-11T00:27:00.475 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-pyasn1. 2021-02-11T00:27:00.492 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-02-11T00:27:00.533 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-02-11T00:27:00.805 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-jsonpatch. 2021-02-11T00:27:00.820 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-02-11T00:27:00.866 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-11T00:27:00.895 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-02-11T00:27:00.910 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-02-11T00:27:00.962 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-11T00:27:01.168 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-02-11T00:27:01.184 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../079-ceph-mgr-rook_17.0.0-691-g597c1051-1bionic_all.deb ... 2021-02-11T00:27:01.235 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-mgr-rook (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:01.315 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-rsa. 2021-02-11T00:27:01.328 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-02-11T00:27:01.373 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-02-11T00:27:01.588 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libonig4:amd64. 2021-02-11T00:27:01.603 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-02-11T00:27:01.701 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-02-11T00:27:01.715 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-02-11T00:27:01.731 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-02-11T00:27:01.759 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-11T00:27:02.222 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-oauth2client. 2021-02-11T00:27:02.238 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-02-11T00:27:02.280 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-02-11T00:27:02.468 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libjq1:amd64. 2021-02-11T00:27:02.482 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-02-11T00:27:02.526 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-02-11T00:27:02.600 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-setuptools. 2021-02-11T00:27:02.616 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-02-11T00:27:02.659 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-02-11T00:27:02.837 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package jq. 2021-02-11T00:27:02.854 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-02-11T00:27:02.903 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-02-11T00:27:03.054 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-websocket. 2021-02-11T00:27:03.067 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-02-11T00:27:03.112 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-02-11T00:27:03.156 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package socat. 2021-02-11T00:27:03.172 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-02-11T00:27:03.214 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-02-11T00:27:03.440 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-kubernetes. 2021-02-11T00:27:03.455 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-02-11T00:27:03.498 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-11T00:27:03.609 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-02-11T00:27:03.624 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-02-11T00:27:03.668 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-11T00:27:03.945 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package xmlstarlet. 2021-02-11T00:27:03.952 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-02-11T00:27:03.959 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-02-11T00:27:03.969 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_17.0.0-691-g597c1051-1bionic_all.deb ... 2021-02-11T00:27:04.003 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-02-11T00:27:04.010 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:04.355 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-json-pointer. 2021-02-11T00:27:04.370 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-02-11T00:27:04.374 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package ceph-test. 2021-02-11T00:27:04.389 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../086-ceph-test_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:27:04.413 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-02-11T00:27:04.433 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking ceph-test (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:04.767 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-jsonpatch. 2021-02-11T00:27:04.782 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-02-11T00:27:04.845 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-11T00:27:05.161 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-02-11T00:27:05.175 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../079-ceph-mgr-rook_17.0.0-691-g597c1051-1bionic_all.deb ... 2021-02-11T00:27:05.219 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking ceph-mgr-rook (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:05.575 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package libonig4:amd64. 2021-02-11T00:27:05.590 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-02-11T00:27:05.719 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-02-11T00:27:06.072 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package libjq1:amd64. 2021-02-11T00:27:06.900 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-02-11T00:27:06.901 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-02-11T00:27:06.901 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package jq. 2021-02-11T00:27:06.901 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-02-11T00:27:06.901 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-02-11T00:27:06.901 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package socat. 2021-02-11T00:27:06.901 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-02-11T00:27:06.901 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-02-11T00:27:07.222 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-02-11T00:27:07.238 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-02-11T00:27:07.280 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-11T00:27:07.558 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package xmlstarlet. 2021-02-11T00:27:07.574 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-02-11T00:27:07.616 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-02-11T00:27:07.946 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package ceph-test. 2021-02-11T00:27:07.963 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../086-ceph-test_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:27:08.003 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking ceph-test (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:08.012 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package libcephfs-dev. 2021-02-11T00:27:08.027 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../087-libcephfs-dev_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:27:08.070 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking libcephfs-dev (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:08.390 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package nvme-cli. 2021-02-11T00:27:08.404 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-02-11T00:27:08.448 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-02-11T00:27:08.777 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python-attr. 2021-02-11T00:27:08.793 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-02-11T00:27:08.835 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python-attr (17.4.0-2) ... 2021-02-11T00:27:09.121 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python-funcsigs. 2021-02-11T00:27:09.136 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-02-11T00:27:09.188 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-02-11T00:27:09.507 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python-pluggy. 2021-02-11T00:27:09.521 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-02-11T00:27:09.573 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-02-11T00:27:09.877 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python-py. 2021-02-11T00:27:09.892 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-02-11T00:27:09.935 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python-py (1.5.2-1) ... 2021-02-11T00:27:10.297 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python-pytest. 2021-02-11T00:27:10.312 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-02-11T00:27:10.355 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-02-11T00:27:10.708 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-attr. 2021-02-11T00:27:10.722 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-02-11T00:27:10.766 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-02-11T00:27:11.061 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-cycler. 2021-02-11T00:27:11.077 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-02-11T00:27:11.119 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-02-11T00:27:11.383 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package libcephfs-dev. 2021-02-11T00:27:11.397 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-webencodings. 2021-02-11T00:27:11.398 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../087-libcephfs-dev_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:27:11.414 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-02-11T00:27:11.441 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking libcephfs-dev (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:11.455 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-02-11T00:27:11.727 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package nvme-cli. 2021-02-11T00:27:11.742 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-02-11T00:27:11.758 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-html5lib. 2021-02-11T00:27:11.772 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-02-11T00:27:11.786 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-02-11T00:27:11.816 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-02-11T00:27:12.140 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python-attr. 2021-02-11T00:27:12.155 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-02-11T00:27:12.198 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python-attr (17.4.0-2) ... 2021-02-11T00:27:12.210 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-02-11T00:27:12.227 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-02-11T00:27:12.277 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-11T00:27:12.543 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python-funcsigs. 2021-02-11T00:27:12.558 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-02-11T00:27:12.601 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-02-11T00:27:12.792 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pyparsing. 2021-02-11T00:27:12.808 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-02-11T00:27:12.847 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-11T00:27:12.887 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python-pluggy. 2021-02-11T00:27:12.901 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-02-11T00:27:12.954 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-02-11T00:27:13.166 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-tz. 2021-02-11T00:27:13.181 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-02-11T00:27:13.225 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-tz (2018.3-2) ... 2021-02-11T00:27:13.273 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python-py. 2021-02-11T00:27:13.287 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-02-11T00:27:13.332 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python-py (1.5.2-1) ... 2021-02-11T00:27:13.536 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-matplotlib. 2021-02-11T00:27:13.550 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-02-11T00:27:13.594 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-11T00:27:13.652 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python-pytest. 2021-02-11T00:27:13.667 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-02-11T00:27:13.710 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-02-11T00:27:14.031 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-attr. 2021-02-11T00:27:14.047 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-02-11T00:27:14.089 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-02-11T00:27:14.276 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-olefile. 2021-02-11T00:27:14.293 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-02-11T00:27:14.333 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-02-11T00:27:14.400 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-cycler. 2021-02-11T00:27:14.414 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-02-11T00:27:14.458 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-02-11T00:27:14.628 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pastescript. 2021-02-11T00:27:14.645 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-02-11T00:27:14.686 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-02-11T00:27:14.761 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-webencodings. 2021-02-11T00:27:14.777 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-02-11T00:27:14.819 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-02-11T00:27:15.049 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pil:amd64. 2021-02-11T00:27:15.063 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-02-11T00:27:15.107 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-11T00:27:15.141 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-html5lib. 2021-02-11T00:27:15.155 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-02-11T00:27:15.216 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-02-11T00:27:15.468 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pluggy. 2021-02-11T00:27:15.484 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-02-11T00:27:15.526 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-02-11T00:27:15.561 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-02-11T00:27:15.579 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-02-11T00:27:15.620 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-11T00:27:15.863 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-py. 2021-02-11T00:27:15.877 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-02-11T00:27:15.929 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-py (1.5.2-1) ... 2021-02-11T00:27:16.082 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-pyparsing. 2021-02-11T00:27:16.097 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-02-11T00:27:16.140 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-11T00:27:16.274 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pyinotify. 2021-02-11T00:27:16.289 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-02-11T00:27:16.332 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-02-11T00:27:16.443 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-tz. 2021-02-11T00:27:16.460 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-02-11T00:27:16.501 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-tz (2018.3-2) ... 2021-02-11T00:27:16.643 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-pytest. 2021-02-11T00:27:16.659 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-02-11T00:27:16.701 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-02-11T00:27:16.804 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-matplotlib. 2021-02-11T00:27:16.819 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-02-11T00:27:16.862 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-11T00:27:17.063 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-simplejson. 2021-02-11T00:27:17.077 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-02-11T00:27:17.130 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-02-11T00:27:17.466 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package python3-tk:amd64. 2021-02-11T00:27:17.481 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-02-11T00:27:17.524 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-11T00:27:17.526 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-olefile. 2021-02-11T00:27:17.544 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-02-11T00:27:17.584 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-02-11T00:27:17.896 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-pastescript. 2021-02-11T00:27:17.912 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-02-11T00:27:17.954 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-02-11T00:27:17.961 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package radosgw. 2021-02-11T00:27:17.978 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../111-radosgw_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:27:18.019 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking radosgw (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:18.342 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-pil:amd64. 2021-02-11T00:27:18.358 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-02-11T00:27:18.391 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-11T00:27:18.761 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-pluggy. 2021-02-11T00:27:18.779 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-02-11T00:27:18.819 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-02-11T00:27:19.081 INFO:teuthology.orchestra.run.smithi057.stdout:Selecting previously unselected package rbd-fuse. 2021-02-11T00:27:19.096 INFO:teuthology.orchestra.run.smithi057.stdout:Preparing to unpack .../112-rbd-fuse_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:27:19.122 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-py. 2021-02-11T00:27:19.138 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-02-11T00:27:19.139 INFO:teuthology.orchestra.run.smithi057.stdout:Unpacking rbd-fuse (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:19.181 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-py (1.5.2-1) ... 2021-02-11T00:27:19.441 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-02-11T00:27:19.509 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-pyinotify. 2021-02-11T00:27:19.523 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-02-11T00:27:19.567 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-02-11T00:27:19.761 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-11T00:27:19.895 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-pytest. 2021-02-11T00:27:19.910 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-02-11T00:27:19.953 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-02-11T00:27:20.145 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-11T00:27:20.296 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-11T00:27:20.324 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-simplejson. 2021-02-11T00:27:20.339 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-02-11T00:27:20.390 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-02-11T00:27:20.727 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package python3-tk:amd64. 2021-02-11T00:27:20.740 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-02-11T00:27:20.802 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-11T00:27:20.924 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-11T00:27:21.239 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package radosgw. 2021-02-11T00:27:21.256 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../111-radosgw_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:27:21.297 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking radosgw (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:21.317 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-02-11T00:27:21.601 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-webencodings (0.5-2) ... 2021-02-11T00:27:21.911 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python-attr (17.4.0-2) ... 2021-02-11T00:27:22.112 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-02-11T00:27:22.272 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-py (1.5.2-1) ... 2021-02-11T00:27:22.350 INFO:teuthology.orchestra.run.smithi095.stdout:Selecting previously unselected package rbd-fuse. 2021-02-11T00:27:22.367 INFO:teuthology.orchestra.run.smithi095.stdout:Preparing to unpack .../112-rbd-fuse_17.0.0-691-g597c1051-1bionic_amd64.deb ... 2021-02-11T00:27:22.408 INFO:teuthology.orchestra.run.smithi095.stdout:Unpacking rbd-fuse (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:22.615 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-02-11T00:27:22.716 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-02-11T00:27:22.941 INFO:teuthology.orchestra.run.smithi057.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-02-11T00:27:23.021 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-02-11T00:27:23.226 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-02-11T00:27:23.381 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-02-11T00:27:23.516 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-02-11T00:27:23.527 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-ceph-argparse (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:23.845 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-02-11T00:27:24.143 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-02-11T00:27:24.154 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-02-11T00:27:24.305 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-11T00:27:24.528 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-02-11T00:27:24.804 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-webencodings (0.5-2) ... 2021-02-11T00:27:25.097 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python-attr (17.4.0-2) ... 2021-02-11T00:27:25.275 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-02-11T00:27:25.315 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-02-11T00:27:25.441 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-py (1.5.2-1) ... 2021-02-11T00:27:25.592 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-02-11T00:27:25.784 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-02-11T00:27:25.893 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python-py (1.5.2-1) ... 2021-02-11T00:27:26.136 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-02-11T00:27:26.144 INFO:teuthology.orchestra.run.smithi095.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-02-11T00:27:26.454 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-02-11T00:27:26.487 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-02-11T00:27:26.586 INFO:teuthology.orchestra.run.smithi057.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-02-11T00:27:26.747 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-ceph-argparse (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:26.814 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-02-11T00:27:26.940 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-02-11T00:27:27.056 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-02-11T00:27:27.141 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-02-11T00:27:27.366 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-02-11T00:27:27.434 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-11T00:27:27.761 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-02-11T00:27:27.900 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-02-11T00:27:28.054 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-02-11T00:27:28.447 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-02-11T00:27:28.765 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-02-11T00:27:28.820 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-02-11T00:27:29.066 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up cephadm (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:29.138 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-02-11T00:27:29.422 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python-py (1.5.2-1) ... 2021-02-11T00:27:29.657 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-02-11T00:27:29.700 INFO:teuthology.orchestra.run.smithi057.stdout:Adding system user cephadm....done 2021-02-11T00:27:29.881 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-02-11T00:27:30.000 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-02-11T00:27:30.090 INFO:teuthology.orchestra.run.smithi095.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-02-11T00:27:30.182 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-11T00:27:30.308 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-02-11T00:27:30.335 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-02-11T00:27:30.451 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-02-11T00:27:30.461 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-02-11T00:27:30.654 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-02-11T00:27:30.802 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-02-11T00:27:30.946 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-02-11T00:27:31.111 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-02-11T00:27:31.238 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-02-11T00:27:31.264 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-02-11T00:27:31.566 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-ceph-common (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:31.583 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-02-11T00:27:31.893 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-monotonic (1.1-2) ... 2021-02-11T00:27:31.959 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-02-11T00:27:32.194 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-11T00:27:32.319 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-02-11T00:27:32.320 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-11T00:27:32.604 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up cephadm (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:32.671 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-02-11T00:27:33.255 INFO:teuthology.orchestra.run.smithi095.stdout:Adding system user cephadm....done 2021-02-11T00:27:33.306 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-11T00:27:33.405 INFO:teuthology.orchestra.run.smithi057.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-02-11T00:27:33.429 INFO:teuthology.orchestra.run.smithi057.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-02-11T00:27:33.444 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-02-11T00:27:33.675 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-02-11T00:27:33.728 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-02-11T00:27:33.854 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-02-11T00:27:33.977 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-11T00:27:33.997 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-02-11T00:27:34.269 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-11T00:27:34.349 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-02-11T00:27:34.612 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-attr (17.4.0-2) ... 2021-02-11T00:27:34.650 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-02-11T00:27:34.776 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-02-11T00:27:34.932 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-02-11T00:27:35.119 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-ceph-common (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:35.125 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libjaeger (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:35.251 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-11T00:27:35.437 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-monotonic (1.1-2) ... 2021-02-11T00:27:35.661 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-02-11T00:27:35.729 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-02-11T00:27:35.751 INFO:teuthology.orchestra.run.smithi057.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-02-11T00:27:35.856 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-02-11T00:27:36.005 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up liboath0 (2.6.1-1) ... 2021-02-11T00:27:36.199 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-02-11T00:27:36.199 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-11T00:27:36.325 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-11T00:27:36.542 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-02-11T00:27:36.577 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-02-11T00:27:36.632 INFO:teuthology.orchestra.run.smithi095.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-02-11T00:27:36.672 INFO:teuthology.orchestra.run.smithi095.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-02-11T00:27:36.703 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-02-11T00:27:36.969 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-02-11T00:27:37.290 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-11T00:27:37.290 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-02-11T00:27:37.290 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2021-02-11T00:27:37.541 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-tz (2018.3-2) ... 2021-02-11T00:27:37.563 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-02-11T00:27:37.851 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-02-11T00:27:37.897 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-attr (17.4.0-2) ... 2021-02-11T00:27:38.153 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-02-11T00:27:38.307 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-02-11T00:27:38.463 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-joblib (0.11-1) ... 2021-02-11T00:27:38.510 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up libjaeger (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:38.653 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-02-11T00:27:38.856 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-routes (2.4.1-1) ... 2021-02-11T00:27:39.071 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-02-11T00:27:39.161 INFO:teuthology.orchestra.run.smithi095.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-02-11T00:27:39.174 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-02-11T00:27:39.423 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up liboath0 (2.6.1-1) ... 2021-02-11T00:27:39.475 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-02-11T00:27:39.599 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-02-11T00:27:39.734 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-02-11T00:27:39.785 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-11T00:27:39.994 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-02-11T00:27:40.145 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-02-11T00:27:40.154 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-02-11T00:27:40.472 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up jq (1.5+dfsg-2) ... 2021-02-11T00:27:40.555 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-02-11T00:27:40.623 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-02-11T00:27:40.689 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-02-11T00:27:40.991 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-tz (2018.3-2) ... 2021-02-11T00:27:41.285 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-02-11T00:27:41.577 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-02-11T00:27:41.847 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-11T00:27:41.879 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-joblib (0.11-1) ... 2021-02-11T00:27:42.198 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-02-11T00:27:42.248 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-routes (2.4.1-1) ... 2021-02-11T00:27:42.533 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-02-11T00:27:42.541 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-02-11T00:27:42.659 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-02-11T00:27:42.850 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-02-11T00:27:43.036 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python-pytest (3.3.2-2) ... 2021-02-11T00:27:43.185 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-02-11T00:27:43.296 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-02-11T00:27:43.422 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-02-11T00:27:43.537 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-02-11T00:27:43.564 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up librados2 (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:43.699 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-02-11T00:27:43.838 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up jq (1.5+dfsg-2) ... 2021-02-11T00:27:43.964 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-02-11T00:27:44.017 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libcephfs2 (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:44.143 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libcephfs-dev (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:44.269 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-rados (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:44.395 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-02-11T00:27:44.722 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-02-11T00:27:45.031 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-02-11T00:27:45.137 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-02-11T00:27:45.434 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-11T00:27:45.455 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-02-11T00:27:45.790 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-02-11T00:27:45.916 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-02-11T00:27:46.284 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python-pytest (3.3.2-2) ... 2021-02-11T00:27:46.321 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-02-11T00:27:46.544 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-02-11T00:27:46.686 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-02-11T00:27:46.812 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up librados2 (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:46.938 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-02-11T00:27:47.168 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-02-11T00:27:47.240 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up libcephfs2 (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:47.366 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up libcephfs-dev (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:47.463 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up libradosstriper1 (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:47.492 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-rados (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:47.587 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-cephfs (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:47.618 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-02-11T00:27:47.708 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up librgw2 (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:47.834 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-fuse (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:47.927 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-02-11T00:27:48.056 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-11T00:27:48.056 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-11T00:27:48.229 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-02-11T00:27:48.344 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up librbd1 (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:48.487 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-02-11T00:27:48.577 INFO:teuthology.orchestra.run.smithi057.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-02-11T00:27:48.618 INFO:teuthology.orchestra.run.smithi057.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-02-11T00:27:48.632 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-02-11T00:27:48.932 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-11T00:27:49.493 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-02-11T00:27:49.627 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-mgr-modules-core (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:49.753 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-02-11T00:27:50.096 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-rgw (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:50.222 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up rbd-fuse (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:50.339 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-02-11T00:27:50.373 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up python3-rbd (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:50.508 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-common (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:50.649 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up libradosstriper1 (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:50.775 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-cephfs (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:50.901 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up librgw2 (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:51.036 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up ceph-fuse (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:51.043 INFO:teuthology.orchestra.run.smithi057.stdout:Adding group ceph....done 2021-02-11T00:27:51.268 INFO:teuthology.orchestra.run.smithi095.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-11T00:27:51.269 INFO:teuthology.orchestra.run.smithi095.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-02-11T00:27:51.436 INFO:teuthology.orchestra.run.smithi057.stdout:Adding system user ceph....done 2021-02-11T00:27:51.548 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up librbd1 (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:51.630 INFO:teuthology.orchestra.run.smithi057.stdout:Setting system user ceph properties....done 2021-02-11T00:27:51.674 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-02-11T00:27:51.778 INFO:teuthology.orchestra.run.smithi095.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-02-11T00:27:51.788 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-02-11T00:27:51.821 INFO:teuthology.orchestra.run.smithi095.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-02-11T00:27:52.025 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-02-11T00:27:52.128 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-02-11T00:27:52.344 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up radosgw (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:52.610 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-11T00:27:52.611 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-11T00:27:52.797 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up ceph-mgr-modules-core (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:52.923 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-02-11T00:27:53.071 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-test (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:53.197 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-base (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:53.224 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-rgw (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:53.392 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up rbd-fuse (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:53.518 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up python3-rbd (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:53.544 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-02-11T00:27:53.644 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up ceph-common (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:53.842 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-mgr (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:54.061 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-11T00:27:54.061 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-11T00:27:54.086 INFO:teuthology.orchestra.run.smithi095.stdout:Adding group ceph....done 2021-02-11T00:27:54.335 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:54.439 INFO:teuthology.orchestra.run.smithi095.stdout:Adding system user ceph....done 2021-02-11T00:27:54.479 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-osd (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:54.640 INFO:teuthology.orchestra.run.smithi095.stdout:Setting system user ceph properties....done 2021-02-11T00:27:54.795 INFO:teuthology.orchestra.run.smithi095.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-02-11T00:27:54.933 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-11T00:27:54.933 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-11T00:27:55.017 INFO:teuthology.orchestra.run.smithi095.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-02-11T00:27:55.231 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-mgr-cephadm (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:55.304 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up radosgw (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:55.375 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-mgr-rook (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:55.509 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-mds (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:55.538 INFO:teuthology.orchestra.run.smithi095.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-11T00:27:55.538 INFO:teuthology.orchestra.run.smithi095.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-02-11T00:27:55.733 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-11T00:27:55.733 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-11T00:27:55.906 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up ceph-test (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:56.032 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up ceph-base (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:56.036 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-mon (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:56.251 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-11T00:27:56.251 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-11T00:27:56.333 INFO:teuthology.orchestra.run.smithi095.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-02-11T00:27:56.554 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-mgr-k8sevents (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:56.625 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up ceph-mgr (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:56.655 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph-mgr-dashboard (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:56.773 INFO:teuthology.orchestra.run.smithi057.stdout:Setting up ceph (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:56.845 INFO:teuthology.orchestra.run.smithi095.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-11T00:27:56.845 INFO:teuthology.orchestra.run.smithi095.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-02-11T00:27:56.891 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-02-11T00:27:57.101 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-02-11T00:27:57.118 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:57.261 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up ceph-osd (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:57.726 INFO:teuthology.orchestra.run.smithi095.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-11T00:27:57.726 INFO:teuthology.orchestra.run.smithi095.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-02-11T00:27:58.013 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up ceph-mgr-cephadm (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:58.132 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up ceph-mgr-rook (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:58.250 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up ceph-mds (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:58.651 INFO:teuthology.orchestra.run.smithi095.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-11T00:27:58.651 INFO:teuthology.orchestra.run.smithi095.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-02-11T00:27:58.952 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up ceph-mon (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:59.161 INFO:teuthology.orchestra.run.smithi095.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-11T00:27:59.161 INFO:teuthology.orchestra.run.smithi095.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-02-11T00:27:59.329 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-02-11T00:27:59.382 INFO:teuthology.orchestra.run.smithi057.stdout:ureadahead will be reprofiled on next reboot 2021-02-11T00:27:59.429 INFO:teuthology.orchestra.run.smithi057.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-02-11T00:27:59.453 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up ceph-mgr-k8sevents (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:59.559 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up ceph-mgr-dashboard (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:59.673 INFO:teuthology.orchestra.run.smithi095.stdout:Setting up ceph (17.0.0-691-g597c1051-1bionic) ... 2021-02-11T00:27:59.782 INFO:teuthology.orchestra.run.smithi095.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-02-11T00:27:59.874 INFO:teuthology.orchestra.run.smithi057.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-02-11T00:27:59.878 DEBUG:teuthology.parallel:result is None 2021-02-11T00:27:59.975 INFO:teuthology.orchestra.run.smithi095.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-02-11T00:28:01.994 INFO:teuthology.orchestra.run.smithi095.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-02-11T00:28:02.036 INFO:teuthology.orchestra.run.smithi095.stdout:ureadahead will be reprofiled on next reboot 2021-02-11T00:28:02.094 INFO:teuthology.orchestra.run.smithi095.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-02-11T00:28:02.536 INFO:teuthology.orchestra.run.smithi095.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-02-11T00:28:02.538 DEBUG:teuthology.parallel:result is None 2021-02-11T00:28:02.539 DEBUG:teuthology.orchestra.run.smithi057:> sudo lsb_release -is 2021-02-11T00:28:02.596 INFO:teuthology.orchestra.run.smithi057.stdout:Ubuntu 2021-02-11T00:28:02.601 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T00:28:02.602 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=597c1051df9f309aa7d6c6db46dcbc2c3982c6e3 2021-02-11T00:28:02.775 DEBUG:teuthology.orchestra.run.smithi057:> dpkg-query -W -f '${Version}' ceph 2021-02-11T00:28:02.799 INFO:teuthology.orchestra.run.smithi057.stdout:17.0.0-691-g597c1051-1bionic 2021-02-11T00:28:02.800 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-691-g597c1051-1bionic 2021-02-11T00:28:02.800 INFO:teuthology.task.install:The correct ceph version 17.0.0-691-g597c1051-1bionic is installed. 2021-02-11T00:28:02.800 DEBUG:teuthology.orchestra.run.smithi095:> sudo lsb_release -is 2021-02-11T00:28:02.855 INFO:teuthology.orchestra.run.smithi095.stdout:Ubuntu 2021-02-11T00:28:02.861 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-02-11T00:28:02.861 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=597c1051df9f309aa7d6c6db46dcbc2c3982c6e3 2021-02-11T00:28:03.030 DEBUG:teuthology.orchestra.run.smithi095:> dpkg-query -W -f '${Version}' ceph 2021-02-11T00:28:03.054 INFO:teuthology.orchestra.run.smithi095.stdout:17.0.0-691-g597c1051-1bionic 2021-02-11T00:28:03.054 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-691-g597c1051-1bionic 2021-02-11T00:28:03.054 INFO:teuthology.task.install:The correct ceph version 17.0.0-691-g597c1051-1bionic is installed. 2021-02-11T00:28:03.054 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-02-11T00:28:03.055 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:03.055 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-11T00:28:03.070 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:03.070 DEBUG:teuthology.orchestra.run.smithi095:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-11T00:28:03.110 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-02-11T00:28:03.110 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:03.111 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/daemon-helper 2021-02-11T00:28:03.128 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-11T00:28:03.184 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:03.184 DEBUG:teuthology.orchestra.run.smithi095:> sudo dd of=/usr/bin/daemon-helper 2021-02-11T00:28:03.198 DEBUG:teuthology.orchestra.run.smithi095:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-11T00:28:03.257 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-02-11T00:28:03.258 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:03.258 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-11T00:28:03.271 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-11T00:28:03.331 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:03.331 DEBUG:teuthology.orchestra.run.smithi095:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-11T00:28:03.346 DEBUG:teuthology.orchestra.run.smithi095:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-11T00:28:03.402 INFO:teuthology.run_tasks:Running task ceph... 2021-02-11T00:28:03.457 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-02-11T00:28:03.457 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 777 /var/log/ceph 2021-02-11T00:28:03.459 DEBUG:teuthology.orchestra.run.smithi095:> sudo chmod 777 /var/log/ceph 2021-02-11T00:28:03.474 INFO:tasks.ceph:Disabling ceph logrotate... 2021-02-11T00:28:03.474 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/logrotate.d/ceph 2021-02-11T00:28:03.516 DEBUG:teuthology.orchestra.run.smithi095:> sudo rm -f -- /etc/logrotate.d/ceph 2021-02-11T00:28:03.530 INFO:tasks.ceph:Creating extra log directories... 2021-02-11T00:28:03.530 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-02-11T00:28:03.571 DEBUG:teuthology.orchestra.run.smithi095:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-02-11T00:28:03.630 INFO:tasks.ceph:Creating ceph cluster c1... 2021-02-11T00:28:03.630 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-02-11T00:28:03.630 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi/5874431', 'branch': 'wip-datalog-genbacking', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'aemerson@redhat.com', 'first_in_suite': False, 'job_id': '5874431', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-datalog-genbacking'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '597c1051df9f309aa7d6c6db46dcbc2c3982c6e3', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '597c1051df9f309aa7d6c6db46dcbc2c3982c6e3'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-datalog-genbacking', 'sha1': '597c1051df9f309aa7d6c6db46dcbc2c3982c6e3'}}, 'owner': 'scheduled_aemerson@teuthology', 'priority': 68, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '597c1051df9f309aa7d6c6db46dcbc2c3982c6e3', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-datalog-genbacking', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '597c1051df9f309aa7d6c6db46dcbc2c3982c6e3', 'targets': {'smithi057.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKoCvpwFe8apsck2MdrWK3EDZV8EosBNVkB/RSE/yQxHn50Ykjz3HAw8V3iKist9ypyMYwUH6KBS+Nn0x4zE6rbr0psdkO5PoNVvOozmdswDNzLbfFuT3l34PG/m0V/LMMWkTBNdvnxP66WjA8kScX5YLa3TFxCReuQAW8CNPEipAzHVgW3NQhD+M/xR4kwV94O9O/QVEiQsblbbrsas74gf/U58cU8nCWkDiymUUhxl23YvtuxjEfRslH3zoC0L5IMJiW92hCLxccM8HX1ZlO7ZLBiaor4JXRayfSFYwND85bXct1kJ/zjbgIE5ZgnB7FoQHtSZnKeN56W5XOD4Xf', 'smithi095.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj65BMVOnEAISCOlnW3qjVHQt+aIJPq3o/p+le9aqXL2+AQ8VXsBSewIDMWDtIXWmMa//mBFRa9x0oGKXHWPK7w+n23MUBlwQHnTHvPGpczSjnb+PXmwBIZYaTRl7gyl6ts8aecf/FrTM54fhGyMMGYXh3ZIJwuh9NyBMf9BKnRfsBQ6jTkbTKbMKL7XmRisu3EcaelLPM3aiLpuyCYw66OgBdMPvC4Bgvo9+J1lRVdivlw3PuagJn4soMxyAFF6zXJ6DnhAX0PwEsJmK6PrQrBwZ7bFy+lXFq8/bDsGwPrG8cynvH8kMKCjl/VoP1wPKwbg3e8XK/Ci2SQ48VPvMR'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '597c1051df9f309aa7d6c6db46dcbc2c3982c6e3', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2021-02-11_00:10:51', 'tube': 'smithi', 'user': 'aemerson', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.29888'} 2021-02-11T00:28:03.631 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-02-11T00:28:03.633 DEBUG:teuthology.orchestra.run.smithi095:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-02-11T00:28:03.640 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m0777 -- /var/run/ceph 2021-02-11T00:28:03.683 DEBUG:teuthology.orchestra.run.smithi095:> sudo install -d -m0777 -- /var/run/ceph 2021-02-11T00:28:03.698 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:03.698 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/scratch_devs of=/dev/stdout 2021-02-11T00:28:03.743 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-11T00:28:03.743 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-11T00:28:03.792 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-02-11T00:28:03.792 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T00:28:03.792 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 6h/6d Inode: 340 Links: 1 2021-02-11T00:28:03.792 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T00:28:03.792 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2021-02-11 00:24:54.794467961 +0000 2021-02-11T00:28:03.792 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2021-02-11 00:24:54.706511963 +0000 2021-02-11T00:28:03.792 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2021-02-11 00:24:54.706511963 +0000 2021-02-11T00:28:03.792 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: - 2021-02-11T00:28:03.802 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2021-02-11T00:28:03.803 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2021-02-11T00:28:03.803 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000598178 s, 856 kB/s 2021-02-11T00:28:03.806 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-11T00:28:03.856 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-02-11T00:28:03.856 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T00:28:03.856 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 6h/6d Inode: 346 Links: 1 2021-02-11T00:28:03.856 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T00:28:03.856 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2021-02-11 00:24:55.018355956 +0000 2021-02-11T00:28:03.856 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2021-02-11 00:24:54.918405958 +0000 2021-02-11T00:28:03.856 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2021-02-11 00:24:54.918405958 +0000 2021-02-11T00:28:03.856 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: - 2021-02-11T00:28:03.866 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2021-02-11T00:28:03.866 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2021-02-11T00:28:03.866 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000487474 s, 1.1 MB/s 2021-02-11T00:28:03.870 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-11T00:28:03.920 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-02-11T00:28:03.920 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T00:28:03.920 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 6h/6d Inode: 352 Links: 1 2021-02-11T00:28:03.920 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T00:28:03.920 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2021-02-11 00:24:55.258235950 +0000 2021-02-11T00:28:03.920 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2021-02-11 00:24:55.162283953 +0000 2021-02-11T00:28:03.920 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2021-02-11 00:24:55.162283953 +0000 2021-02-11T00:28:03.920 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: - 2021-02-11T00:28:03.930 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2021-02-11T00:28:03.930 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2021-02-11T00:28:03.930 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000294294 s, 1.7 MB/s 2021-02-11T00:28:03.935 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-11T00:28:03.984 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-02-11T00:28:03.984 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T00:28:03.984 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 6h/6d Inode: 358 Links: 1 2021-02-11T00:28:03.984 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T00:28:03.985 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2021-02-11 00:24:55.370179948 +0000 2021-02-11T00:28:03.985 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2021-02-11 00:24:55.366181948 +0000 2021-02-11T00:28:03.985 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2021-02-11 00:24:55.366181948 +0000 2021-02-11T00:28:03.985 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: - 2021-02-11T00:28:03.993 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2021-02-11T00:28:03.993 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2021-02-11T00:28:03.993 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000310063 s, 1.7 MB/s 2021-02-11T00:28:03.997 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T00:28:03.997 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi057.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-11T00:28:03.997 INFO:tasks.ceph:Generating config... 2021-02-11T00:28:03.999 INFO:tasks.ceph:[client] debug rgw = 20 2021-02-11T00:28:03.999 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-02-11T00:28:03.999 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-02-11T00:28:03.999 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T00:28:03.999 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-02-11T00:28:03.999 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-02-11T00:28:03.999 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-02-11T00:28:03.999 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-02-11T00:28:03.999 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-02-11T00:28:03.999 INFO:tasks.ceph:[client] setgroup = ceph 2021-02-11T00:28:03.999 INFO:tasks.ceph:[client] setuser = ceph 2021-02-11T00:28:03.999 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2021-02-11T00:28:03.999 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-02-11T00:28:04.000 INFO:tasks.ceph:[mgr] debug ms = 1 2021-02-11T00:28:04.000 INFO:tasks.ceph:[mon] debug mon = 20 2021-02-11T00:28:04.000 INFO:tasks.ceph:[mon] debug ms = 1 2021-02-11T00:28:04.000 INFO:tasks.ceph:[mon] debug paxos = 20 2021-02-11T00:28:04.000 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2021-02-11T00:28:04.000 INFO:tasks.ceph:[osd] debug filestore = 20 2021-02-11T00:28:04.000 INFO:tasks.ceph:[osd] debug journal = 20 2021-02-11T00:28:04.000 INFO:tasks.ceph:[osd] debug ms = 20 2021-02-11T00:28:04.000 INFO:tasks.ceph:[osd] debug osd = 25 2021-02-11T00:28:04.000 INFO:tasks.ceph:[osd] osd fast shutdown = False 2021-02-11T00:28:04.000 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-02-11T00:28:04.000 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-02-11T00:28:04.000 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-02-11T00:28:04.001 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-02-11T00:28:04.001 INFO:tasks.ceph:Setting up c1.mon.a... 2021-02-11T00:28:04.001 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2021-02-11T00:28:04.069 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/c1.keyring 2021-02-11T00:28:04.071 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2021-02-11T00:28:04.148 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-02-11T00:28:04.203 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.57')] 2021-02-11T00:28:04.203 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.57', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-02-11T00:28:04.204 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:04.204 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-02-11T00:28:04.251 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.57 --print /home/ubuntu/cephtest/c1.monmap 2021-02-11T00:28:04.324 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:28:04.324 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:28:04.324 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2021-02-11T00:28:04.324 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: generated fsid d110339e-4b9e-408b-b214-00afac1a1a23 2021-02-11T00:28:04.325 INFO:teuthology.orchestra.run.smithi057.stdout:epoch 0 2021-02-11T00:28:04.325 INFO:teuthology.orchestra.run.smithi057.stdout:fsid d110339e-4b9e-408b-b214-00afac1a1a23 2021-02-11T00:28:04.325 INFO:teuthology.orchestra.run.smithi057.stdout:last_changed 2021-02-11T00:28:04.327921+0000 2021-02-11T00:28:04.325 INFO:teuthology.orchestra.run.smithi057.stdout:created 2021-02-11T00:28:04.327921+0000 2021-02-11T00:28:04.325 INFO:teuthology.orchestra.run.smithi057.stdout:min_mon_release 0 (unknown) 2021-02-11T00:28:04.325 INFO:teuthology.orchestra.run.smithi057.stdout:election_strategy: 1 2021-02-11T00:28:04.325 INFO:teuthology.orchestra.run.smithi057.stdout:0: [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] mon.a 2021-02-11T00:28:04.325 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2021-02-11T00:28:04.326 DEBUG:teuthology.orchestra.run.smithi057:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-02-11T00:28:04.375 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID d110339e-4b9e-408b-b214-00afac1a1a23... 2021-02-11T00:28:04.376 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-02-11T00:28:04.419 DEBUG:teuthology.orchestra.run.smithi095:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-02-11T00:28:04.450 INFO:teuthology.orchestra.run.smithi057.stdout:[global] 2021-02-11T00:28:04.451 INFO:teuthology.orchestra.run.smithi057.stdout: chdir = "" 2021-02-11T00:28:04.451 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-11T00:28:04.451 INFO:teuthology.orchestra.run.smithi057.stdout: auth supported = cephx 2021-02-11T00:28:04.451 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.451 INFO:teuthology.orchestra.run.smithi057.stdout: filestore xattr use omap = true 2021-02-11T00:28:04.451 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.451 INFO:teuthology.orchestra.run.smithi057.stdout: mon clock drift allowed = 1.000 2021-02-11T00:28:04.451 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.451 INFO:teuthology.orchestra.run.smithi057.stdout: osd crush chooseleaf type = 0 2021-02-11T00:28:04.451 INFO:teuthology.orchestra.run.smithi057.stdout: auth debug = true 2021-02-11T00:28:04.451 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.451 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on old message = true 2021-02-11T00:28:04.452 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on bug = true 2021-02-11T00:28:04.452 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.452 INFO:teuthology.orchestra.run.smithi057.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-11T00:28:04.452 INFO:teuthology.orchestra.run.smithi057.stdout: mon pg warn max object skew = 0 2021-02-11T00:28:04.452 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.452 INFO:teuthology.orchestra.run.smithi057.stdout: # disable pg_autoscaler by default for new pools 2021-02-11T00:28:04.452 INFO:teuthology.orchestra.run.smithi057.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-11T00:28:04.452 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.452 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default size = 2 2021-02-11T00:28:04.452 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.452 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow primary affinity = true 2021-02-11T00:28:04.452 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow pg remap = true 2021-02-11T00:28:04.452 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on legacy crush tunables = false 2021-02-11T00:28:04.453 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on crush straw calc version zero = false 2021-02-11T00:28:04.453 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on no sortbitwise = false 2021-02-11T00:28:04.453 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on osd down out interval zero = false 2021-02-11T00:28:04.453 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on too few osds = false 2021-02-11T00:28:04.453 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-11T00:28:04.453 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-11T00:28:04.453 INFO:teuthology.orchestra.run.smithi057.stdout: mon_allow_pool_size_one = true 2021-02-11T00:28:04.453 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.453 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-11T00:28:04.453 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.453 INFO:teuthology.orchestra.run.smithi057.stdout: osd default data pool replay window = 5 2021-02-11T00:28:04.453 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.453 INFO:teuthology.orchestra.run.smithi057.stdout: mon allow pool delete = true 2021-02-11T00:28:04.454 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.454 INFO:teuthology.orchestra.run.smithi057.stdout: mon cluster log file level = debug 2021-02-11T00:28:04.454 INFO:teuthology.orchestra.run.smithi057.stdout: debug asserts on shutdown = true 2021-02-11T00:28:04.454 INFO:teuthology.orchestra.run.smithi057.stdout: mon health detail to clog = false 2021-02-11T00:28:04.454 INFO:teuthology.orchestra.run.smithi057.stdout: mon host = 172.21.15.57 2021-02-11T00:28:04.454 INFO:teuthology.orchestra.run.smithi057.stdout: osd heartbeat grace = 40 2021-02-11T00:28:04.454 INFO:teuthology.orchestra.run.smithi057.stdout: fsid = d110339e-4b9e-408b-b214-00afac1a1a23 2021-02-11T00:28:04.454 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.454 INFO:teuthology.orchestra.run.smithi057.stdout:[osd] 2021-02-11T00:28:04.455 INFO:teuthology.orchestra.run.smithi057.stdout: osd journal size = 100 2021-02-11T00:28:04.455 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.455 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub load threshold = 5.0 2021-02-11T00:28:04.455 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub max interval = 600 2021-02-11T00:28:04.455 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.455 INFO:teuthology.orchestra.run.smithi057.stdout: osd recover clone overlap = true 2021-02-11T00:28:04.455 INFO:teuthology.orchestra.run.smithi057.stdout: osd recovery max chunk = 1048576 2021-02-11T00:28:04.455 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.455 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug shutdown = true 2021-02-11T00:28:04.455 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug op order = true 2021-02-11T00:28:04.455 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug verify stray on activate = true 2021-02-11T00:28:04.455 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.455 INFO:teuthology.orchestra.run.smithi057.stdout: osd open classes on start = true 2021-02-11T00:28:04.455 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug pg log writeout = true 2021-02-11T00:28:04.455 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.456 INFO:teuthology.orchestra.run.smithi057.stdout: osd deep scrub update digest min age = 30 2021-02-11T00:28:04.456 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.456 INFO:teuthology.orchestra.run.smithi057.stdout: osd map max advance = 10 2021-02-11T00:28:04.456 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.456 INFO:teuthology.orchestra.run.smithi057.stdout: journal zero on create = true 2021-02-11T00:28:04.456 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.456 INFO:teuthology.orchestra.run.smithi057.stdout: filestore ondisk finisher threads = 3 2021-02-11T00:28:04.456 INFO:teuthology.orchestra.run.smithi057.stdout: filestore apply finisher threads = 3 2021-02-11T00:28:04.456 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.456 INFO:teuthology.orchestra.run.smithi057.stdout: bdev debug aio = true 2021-02-11T00:28:04.456 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug misdirected ops = true 2021-02-11T00:28:04.456 INFO:teuthology.orchestra.run.smithi057.stdout: debug filestore = 20 2021-02-11T00:28:04.456 INFO:teuthology.orchestra.run.smithi057.stdout: debug journal = 20 2021-02-11T00:28:04.457 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 20 2021-02-11T00:28:04.457 INFO:teuthology.orchestra.run.smithi057.stdout: debug osd = 25 2021-02-11T00:28:04.457 INFO:teuthology.orchestra.run.smithi057.stdout: osd fast shutdown = False 2021-02-11T00:28:04.457 INFO:teuthology.orchestra.run.smithi057.stdout: rocksdb delete range threshold = 0 2021-02-11T00:28:04.457 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.457 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr] 2021-02-11T00:28:04.457 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2021-02-11T00:28:04.457 INFO:teuthology.orchestra.run.smithi057.stdout: debug mgr = 20 2021-02-11T00:28:04.457 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2021-02-11T00:28:04.457 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2021-02-11T00:28:04.457 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2021-02-11T00:28:04.457 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2021-02-11T00:28:04.458 INFO:teuthology.orchestra.run.smithi057.stdout: mgr/telemetry/nag = false 2021-02-11T00:28:04.458 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.458 INFO:teuthology.orchestra.run.smithi057.stdout:[mon] 2021-02-11T00:28:04.458 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2021-02-11T00:28:04.458 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2021-02-11T00:28:04.458 INFO:teuthology.orchestra.run.smithi057.stdout: debug paxos = 20 2021-02-11T00:28:04.458 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2021-02-11T00:28:04.458 INFO:teuthology.orchestra.run.smithi057.stdout: mon data avail warn = 5 2021-02-11T00:28:04.458 INFO:teuthology.orchestra.run.smithi057.stdout: mon mgr mkfs grace = 240 2021-02-11T00:28:04.458 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2021-02-11T00:28:04.458 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd reporter subtree level = osd 2021-02-11T00:28:04.458 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd prime pg temp = true 2021-02-11T00:28:04.459 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2021-02-11T00:28:04.459 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd crush smoke test = False 2021-02-11T00:28:04.459 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:04.459 INFO:teuthology.orchestra.run.smithi057.stdout:[client] 2021-02-11T00:28:04.459 INFO:teuthology.orchestra.run.smithi057.stdout: rgw cache enabled = true 2021-02-11T00:28:04.459 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable ops log = true 2021-02-11T00:28:04.459 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable usage log = true 2021-02-11T00:28:04.459 INFO:teuthology.orchestra.run.smithi057.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-11T00:28:04.459 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-11T00:28:04.459 INFO:teuthology.orchestra.run.smithi057.stdout: debug rgw = 20 2021-02-11T00:28:04.459 INFO:teuthology.orchestra.run.smithi057.stdout: rgw crypt require ssl = False 2021-02-11T00:28:04.459 INFO:teuthology.orchestra.run.smithi057.stdout: rgw crypt s3 kms backend = testing 2021-02-11T00:28:04.459 INFO:teuthology.orchestra.run.smithi057.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T00:28:04.459 INFO:teuthology.orchestra.run.smithi057.stdout: rgw curl low speed time = 300 2021-02-11T00:28:04.460 INFO:teuthology.orchestra.run.smithi057.stdout: rgw data log num shards = 4 2021-02-11T00:28:04.460 INFO:teuthology.orchestra.run.smithi057.stdout: rgw md log max shards = 4 2021-02-11T00:28:04.460 INFO:teuthology.orchestra.run.smithi057.stdout: rgw sync log trim interval = 0 2021-02-11T00:28:04.460 INFO:teuthology.orchestra.run.smithi057.stdout: rgw sync obj etag verify = True 2021-02-11T00:28:04.460 INFO:teuthology.orchestra.run.smithi057.stdout: setgroup = ceph 2021-02-11T00:28:04.460 INFO:teuthology.orchestra.run.smithi057.stdout: setuser = ceph 2021-02-11T00:28:04.460 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.a] 2021-02-11T00:28:04.460 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.0] 2021-02-11T00:28:04.460 INFO:teuthology.orchestra.run.smithi057.stdout: osd_max_omap_entries_per_request = 10 2021-02-11T00:28:04.460 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.1] 2021-02-11T00:28:04.460 INFO:teuthology.orchestra.run.smithi057.stdout: osd_max_omap_entries_per_request = 1000 2021-02-11T00:28:04.460 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.2] 2021-02-11T00:28:04.460 INFO:teuthology.orchestra.run.smithi057.stdout: osd_max_omap_entries_per_request = 10000 2021-02-11T00:28:04.462 INFO:teuthology.orchestra.run.smithi095.stdout:[global] 2021-02-11T00:28:04.462 INFO:teuthology.orchestra.run.smithi095.stdout: chdir = "" 2021-02-11T00:28:04.462 INFO:teuthology.orchestra.run.smithi095.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-11T00:28:04.462 INFO:teuthology.orchestra.run.smithi095.stdout: auth supported = cephx 2021-02-11T00:28:04.462 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.462 INFO:teuthology.orchestra.run.smithi095.stdout: filestore xattr use omap = true 2021-02-11T00:28:04.462 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.462 INFO:teuthology.orchestra.run.smithi095.stdout: mon clock drift allowed = 1.000 2021-02-11T00:28:04.462 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.463 INFO:teuthology.orchestra.run.smithi095.stdout: osd crush chooseleaf type = 0 2021-02-11T00:28:04.463 INFO:teuthology.orchestra.run.smithi095.stdout: auth debug = true 2021-02-11T00:28:04.463 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.463 INFO:teuthology.orchestra.run.smithi095.stdout: ms die on old message = true 2021-02-11T00:28:04.463 INFO:teuthology.orchestra.run.smithi095.stdout: ms die on bug = true 2021-02-11T00:28:04.463 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.463 INFO:teuthology.orchestra.run.smithi095.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-11T00:28:04.463 INFO:teuthology.orchestra.run.smithi095.stdout: mon pg warn max object skew = 0 2021-02-11T00:28:04.463 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.463 INFO:teuthology.orchestra.run.smithi095.stdout: # disable pg_autoscaler by default for new pools 2021-02-11T00:28:04.463 INFO:teuthology.orchestra.run.smithi095.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-11T00:28:04.463 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.464 INFO:teuthology.orchestra.run.smithi095.stdout: osd pool default size = 2 2021-02-11T00:28:04.464 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.464 INFO:teuthology.orchestra.run.smithi095.stdout: mon osd allow primary affinity = true 2021-02-11T00:28:04.464 INFO:teuthology.orchestra.run.smithi095.stdout: mon osd allow pg remap = true 2021-02-11T00:28:04.464 INFO:teuthology.orchestra.run.smithi095.stdout: mon warn on legacy crush tunables = false 2021-02-11T00:28:04.464 INFO:teuthology.orchestra.run.smithi095.stdout: mon warn on crush straw calc version zero = false 2021-02-11T00:28:04.464 INFO:teuthology.orchestra.run.smithi095.stdout: mon warn on no sortbitwise = false 2021-02-11T00:28:04.464 INFO:teuthology.orchestra.run.smithi095.stdout: mon warn on osd down out interval zero = false 2021-02-11T00:28:04.464 INFO:teuthology.orchestra.run.smithi095.stdout: mon warn on too few osds = false 2021-02-11T00:28:04.464 INFO:teuthology.orchestra.run.smithi095.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-11T00:28:04.464 INFO:teuthology.orchestra.run.smithi095.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-11T00:28:04.464 INFO:teuthology.orchestra.run.smithi095.stdout: mon_allow_pool_size_one = true 2021-02-11T00:28:04.464 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.465 INFO:teuthology.orchestra.run.smithi095.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-11T00:28:04.465 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.465 INFO:teuthology.orchestra.run.smithi095.stdout: osd default data pool replay window = 5 2021-02-11T00:28:04.465 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.465 INFO:teuthology.orchestra.run.smithi095.stdout: mon allow pool delete = true 2021-02-11T00:28:04.465 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.465 INFO:teuthology.orchestra.run.smithi095.stdout: mon cluster log file level = debug 2021-02-11T00:28:04.465 INFO:teuthology.orchestra.run.smithi095.stdout: debug asserts on shutdown = true 2021-02-11T00:28:04.465 INFO:teuthology.orchestra.run.smithi095.stdout: mon health detail to clog = false 2021-02-11T00:28:04.465 INFO:teuthology.orchestra.run.smithi095.stdout: mon host = 172.21.15.57 2021-02-11T00:28:04.465 INFO:teuthology.orchestra.run.smithi095.stdout: osd heartbeat grace = 40 2021-02-11T00:28:04.465 INFO:teuthology.orchestra.run.smithi095.stdout: fsid = d110339e-4b9e-408b-b214-00afac1a1a23 2021-02-11T00:28:04.466 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.466 INFO:teuthology.orchestra.run.smithi095.stdout:[osd] 2021-02-11T00:28:04.466 INFO:teuthology.orchestra.run.smithi095.stdout: osd journal size = 100 2021-02-11T00:28:04.466 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.466 INFO:teuthology.orchestra.run.smithi095.stdout: osd scrub load threshold = 5.0 2021-02-11T00:28:04.466 INFO:teuthology.orchestra.run.smithi095.stdout: osd scrub max interval = 600 2021-02-11T00:28:04.466 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.466 INFO:teuthology.orchestra.run.smithi095.stdout: osd recover clone overlap = true 2021-02-11T00:28:04.466 INFO:teuthology.orchestra.run.smithi095.stdout: osd recovery max chunk = 1048576 2021-02-11T00:28:04.466 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.466 INFO:teuthology.orchestra.run.smithi095.stdout: osd debug shutdown = true 2021-02-11T00:28:04.466 INFO:teuthology.orchestra.run.smithi095.stdout: osd debug op order = true 2021-02-11T00:28:04.466 INFO:teuthology.orchestra.run.smithi095.stdout: osd debug verify stray on activate = true 2021-02-11T00:28:04.467 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.467 INFO:teuthology.orchestra.run.smithi095.stdout: osd open classes on start = true 2021-02-11T00:28:04.467 INFO:teuthology.orchestra.run.smithi095.stdout: osd debug pg log writeout = true 2021-02-11T00:28:04.467 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.467 INFO:teuthology.orchestra.run.smithi095.stdout: osd deep scrub update digest min age = 30 2021-02-11T00:28:04.467 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.467 INFO:teuthology.orchestra.run.smithi095.stdout: osd map max advance = 10 2021-02-11T00:28:04.467 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.467 INFO:teuthology.orchestra.run.smithi095.stdout: journal zero on create = true 2021-02-11T00:28:04.467 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.467 INFO:teuthology.orchestra.run.smithi095.stdout: filestore ondisk finisher threads = 3 2021-02-11T00:28:04.467 INFO:teuthology.orchestra.run.smithi095.stdout: filestore apply finisher threads = 3 2021-02-11T00:28:04.467 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.467 INFO:teuthology.orchestra.run.smithi095.stdout: bdev debug aio = true 2021-02-11T00:28:04.468 INFO:teuthology.orchestra.run.smithi095.stdout: osd debug misdirected ops = true 2021-02-11T00:28:04.468 INFO:teuthology.orchestra.run.smithi095.stdout: debug filestore = 20 2021-02-11T00:28:04.468 INFO:teuthology.orchestra.run.smithi095.stdout: debug journal = 20 2021-02-11T00:28:04.468 INFO:teuthology.orchestra.run.smithi095.stdout: debug ms = 20 2021-02-11T00:28:04.468 INFO:teuthology.orchestra.run.smithi095.stdout: debug osd = 25 2021-02-11T00:28:04.468 INFO:teuthology.orchestra.run.smithi095.stdout: osd fast shutdown = False 2021-02-11T00:28:04.468 INFO:teuthology.orchestra.run.smithi095.stdout: rocksdb delete range threshold = 0 2021-02-11T00:28:04.468 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.468 INFO:teuthology.orchestra.run.smithi095.stdout:[mgr] 2021-02-11T00:28:04.468 INFO:teuthology.orchestra.run.smithi095.stdout: debug ms = 1 2021-02-11T00:28:04.468 INFO:teuthology.orchestra.run.smithi095.stdout: debug mgr = 20 2021-02-11T00:28:04.468 INFO:teuthology.orchestra.run.smithi095.stdout: debug mon = 20 2021-02-11T00:28:04.468 INFO:teuthology.orchestra.run.smithi095.stdout: debug auth = 20 2021-02-11T00:28:04.468 INFO:teuthology.orchestra.run.smithi095.stdout: mon reweight min pgs per osd = 4 2021-02-11T00:28:04.469 INFO:teuthology.orchestra.run.smithi095.stdout: mon reweight min bytes per osd = 10 2021-02-11T00:28:04.469 INFO:teuthology.orchestra.run.smithi095.stdout: mgr/telemetry/nag = false 2021-02-11T00:28:04.469 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.469 INFO:teuthology.orchestra.run.smithi095.stdout:[mon] 2021-02-11T00:28:04.469 INFO:teuthology.orchestra.run.smithi095.stdout: debug ms = 1 2021-02-11T00:28:04.469 INFO:teuthology.orchestra.run.smithi095.stdout: debug mon = 20 2021-02-11T00:28:04.469 INFO:teuthology.orchestra.run.smithi095.stdout: debug paxos = 20 2021-02-11T00:28:04.469 INFO:teuthology.orchestra.run.smithi095.stdout: debug auth = 20 2021-02-11T00:28:04.469 INFO:teuthology.orchestra.run.smithi095.stdout: mon data avail warn = 5 2021-02-11T00:28:04.469 INFO:teuthology.orchestra.run.smithi095.stdout: mon mgr mkfs grace = 240 2021-02-11T00:28:04.469 INFO:teuthology.orchestra.run.smithi095.stdout: mon reweight min pgs per osd = 4 2021-02-11T00:28:04.469 INFO:teuthology.orchestra.run.smithi095.stdout: mon osd reporter subtree level = osd 2021-02-11T00:28:04.469 INFO:teuthology.orchestra.run.smithi095.stdout: mon osd prime pg temp = true 2021-02-11T00:28:04.469 INFO:teuthology.orchestra.run.smithi095.stdout: mon reweight min bytes per osd = 10 2021-02-11T00:28:04.469 INFO:teuthology.orchestra.run.smithi095.stdout: mon osd crush smoke test = False 2021-02-11T00:28:04.470 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:04.470 INFO:teuthology.orchestra.run.smithi095.stdout:[client] 2021-02-11T00:28:04.470 INFO:teuthology.orchestra.run.smithi095.stdout: rgw cache enabled = true 2021-02-11T00:28:04.470 INFO:teuthology.orchestra.run.smithi095.stdout: rgw enable ops log = true 2021-02-11T00:28:04.470 INFO:teuthology.orchestra.run.smithi095.stdout: rgw enable usage log = true 2021-02-11T00:28:04.470 INFO:teuthology.orchestra.run.smithi095.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-11T00:28:04.470 INFO:teuthology.orchestra.run.smithi095.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-11T00:28:04.470 INFO:teuthology.orchestra.run.smithi095.stdout: debug rgw = 20 2021-02-11T00:28:04.470 INFO:teuthology.orchestra.run.smithi095.stdout: rgw crypt require ssl = False 2021-02-11T00:28:04.470 INFO:teuthology.orchestra.run.smithi095.stdout: rgw crypt s3 kms backend = testing 2021-02-11T00:28:04.470 INFO:teuthology.orchestra.run.smithi095.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T00:28:04.470 INFO:teuthology.orchestra.run.smithi095.stdout: rgw curl low speed time = 300 2021-02-11T00:28:04.470 INFO:teuthology.orchestra.run.smithi095.stdout: rgw data log num shards = 4 2021-02-11T00:28:04.470 INFO:teuthology.orchestra.run.smithi095.stdout: rgw md log max shards = 4 2021-02-11T00:28:04.471 INFO:teuthology.orchestra.run.smithi095.stdout: rgw sync log trim interval = 0 2021-02-11T00:28:04.471 INFO:teuthology.orchestra.run.smithi095.stdout: rgw sync obj etag verify = True 2021-02-11T00:28:04.471 INFO:teuthology.orchestra.run.smithi095.stdout: setgroup = ceph 2021-02-11T00:28:04.471 INFO:teuthology.orchestra.run.smithi095.stdout: setuser = ceph 2021-02-11T00:28:04.471 INFO:teuthology.orchestra.run.smithi095.stdout:[mon.a] 2021-02-11T00:28:04.471 INFO:teuthology.orchestra.run.smithi095.stdout:[osd.0] 2021-02-11T00:28:04.471 INFO:teuthology.orchestra.run.smithi095.stdout: osd_max_omap_entries_per_request = 10 2021-02-11T00:28:04.471 INFO:teuthology.orchestra.run.smithi095.stdout:[osd.1] 2021-02-11T00:28:04.471 INFO:teuthology.orchestra.run.smithi095.stdout: osd_max_omap_entries_per_request = 1000 2021-02-11T00:28:04.471 INFO:teuthology.orchestra.run.smithi095.stdout:[osd.2] 2021-02-11T00:28:04.471 INFO:teuthology.orchestra.run.smithi095.stdout: osd_max_omap_entries_per_request = 10000 2021-02-11T00:28:04.472 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2021-02-11T00:28:04.472 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2021-02-11T00:28:04.551 INFO:tasks.ceph:Copying monmap to all nodes... 2021-02-11T00:28:04.551 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:04.551 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2021-02-11T00:28:04.555 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:04.555 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2021-02-11T00:28:04.603 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi057.front.sepia.ceph.com 2021-02-11T00:28:04.603 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:04.603 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/c1.keyring 2021-02-11T00:28:04.603 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-02-11T00:28:04.671 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:04.671 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-02-11T00:28:04.719 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi095.front.sepia.ceph.com 2021-02-11T00:28:04.719 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:04.720 DEBUG:teuthology.orchestra.run.smithi095:> sudo dd of=/etc/ceph/c1.keyring 2021-02-11T00:28:04.720 DEBUG:teuthology.orchestra.run.smithi095:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-02-11T00:28:04.743 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:04.743 DEBUG:teuthology.orchestra.run.smithi095:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-02-11T00:28:04.793 INFO:tasks.ceph:Setting up mon nodes... 2021-02-11T00:28:04.794 INFO:tasks.ceph:Setting up mgr nodes... 2021-02-11T00:28:04.794 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2021-02-11T00:28:04.834 INFO:teuthology.orchestra.run.smithi057.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2021-02-11T00:28:04.837 INFO:tasks.ceph:Setting up mds nodes... 2021-02-11T00:28:04.837 INFO:tasks.ceph_client:Setting up client nodes... 2021-02-11T00:28:04.837 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2021-02-11T00:28:04.910 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/c1.client.0.keyring 2021-02-11T00:28:04.922 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2021-02-11T00:28:05.000 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/c1.client.1.keyring 2021-02-11T00:28:05.011 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-02-11T00:28:05.012 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi057.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-11T00:28:05.012 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2021-02-11T00:28:05.067 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T00:28:05.067 INFO:tasks.ceph:role: c1.osd.0 2021-02-11T00:28:05.067 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi057.front.sepia.ceph.com 2021-02-11T00:28:05.067 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-02-11T00:28:06.449 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T00:28:06.449 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T00:28:06.449 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T00:28:06.449 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T00:28:06.449 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2021-02-11T00:28:06.450 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T00:28:06.450 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T00:28:06.450 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T00:28:06.450 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T00:28:06.451 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2021-02-11T00:28:06.451 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2021-02-11T00:28:06.473 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2021-02-11T00:28:06.524 INFO:teuthology.orchestra.run.smithi057.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T00:28:06.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T00:28:06.525 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2021-02-11T00:28:06.579 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T00:28:06.579 INFO:tasks.ceph:role: c1.osd.1 2021-02-11T00:28:06.579 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi057.front.sepia.ceph.com 2021-02-11T00:28:06.580 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-02-11T00:28:07.962 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T00:28:07.962 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T00:28:07.962 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T00:28:07.962 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T00:28:07.962 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2021-02-11T00:28:07.962 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T00:28:07.962 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T00:28:07.963 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T00:28:07.963 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T00:28:07.963 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2021-02-11T00:28:07.964 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2021-02-11T00:28:07.987 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2021-02-11T00:28:08.043 INFO:teuthology.orchestra.run.smithi057.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T00:28:08.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T00:28:08.044 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2021-02-11T00:28:08.101 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T00:28:08.101 INFO:tasks.ceph:role: c1.osd.2 2021-02-11T00:28:08.101 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi057.front.sepia.ceph.com 2021-02-11T00:28:08.102 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-02-11T00:28:09.489 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T00:28:09.489 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T00:28:09.489 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T00:28:09.489 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T00:28:09.489 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2021-02-11T00:28:09.489 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T00:28:09.490 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T00:28:09.490 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T00:28:09.490 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T00:28:09.491 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2021-02-11T00:28:09.491 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2021-02-11T00:28:09.513 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2021-02-11T00:28:09.568 INFO:teuthology.orchestra.run.smithi057.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T00:28:09.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T00:28:09.569 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2021-02-11T00:28:09.657 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:09.656+0000 7f3a6c03e080 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2021-02-11T00:28:09.657 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:09.656+0000 7f3a6c03e080 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2021-02-11T00:28:09.657 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:09.656+0000 7f3a6c03e080 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-02-11T00:28:09.657 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:09.656+0000 7f3a6c03e080 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-02-11T00:28:09.657 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:09.656+0000 7f3a6c03e080 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-02-11T00:28:09.658 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:09.660+0000 7f3a6c03e080 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2021-02-11T00:28:11.862 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2021-02-11T00:28:11.878 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2021-02-11T00:28:11.959 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:11.960+0000 7fcb4e84e080 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2021-02-11T00:28:11.960 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:11.960+0000 7fcb4e84e080 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2021-02-11T00:28:11.960 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:11.960+0000 7fcb4e84e080 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-02-11T00:28:11.960 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:11.960+0000 7fcb4e84e080 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-02-11T00:28:11.960 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:11.960+0000 7fcb4e84e080 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-02-11T00:28:11.961 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:11.960+0000 7fcb4e84e080 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2021-02-11T00:28:14.186 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2021-02-11T00:28:14.245 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2021-02-11T00:28:14.328 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:14.328+0000 7f7e84c87080 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2021-02-11T00:28:14.329 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:14.328+0000 7f7e84c87080 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2021-02-11T00:28:14.329 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:14.328+0000 7f7e84c87080 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-02-11T00:28:14.329 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:14.328+0000 7f7e84c87080 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-02-11T00:28:14.329 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:14.328+0000 7f7e84c87080 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-02-11T00:28:14.330 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:28:14.332+0000 7f7e84c87080 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2021-02-11T00:28:16.525 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2021-02-11T00:28:16.581 INFO:tasks.ceph:Reading keys from all nodes... 2021-02-11T00:28:16.581 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:16.581 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2021-02-11T00:28:16.640 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:16.640 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2021-02-11T00:28:16.696 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:16.696 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2021-02-11T00:28:16.751 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:16.752 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2021-02-11T00:28:16.808 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:16.808 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2021-02-11T00:28:16.855 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:16.855 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2021-02-11T00:28:16.903 INFO:tasks.ceph:Adding keys to all mons... 2021-02-11T00:28:16.903 DEBUG:teuthology.orchestra.run.smithi057:> sudo tee -a /etc/ceph/c1.keyring 2021-02-11T00:28:16.958 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr.x] 2021-02-11T00:28:16.958 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAUeiRgUnX3MRAAv2trfaKUHW52Q50VC6frqQ== 2021-02-11T00:28:16.959 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.0] 2021-02-11T00:28:16.959 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAZeiRg8xpTJxAA+tYGfNEZza/ZrbfkLUP2sQ== 2021-02-11T00:28:16.959 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.1] 2021-02-11T00:28:16.959 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAbeiRgbatpORAAjzKf7uDqijgybF6XEFO5QQ== 2021-02-11T00:28:16.959 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.2] 2021-02-11T00:28:16.959 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAeeiRgMZXIExAAb0Ba/vivG9yRbupDOvgY0A== 2021-02-11T00:28:16.959 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2021-02-11T00:28:16.959 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAUeiRgCJB4NhAAdgUdPXPqOWjeXxoSM0NW/Q== 2021-02-11T00:28:16.959 INFO:teuthology.orchestra.run.smithi057.stdout:[client.1] 2021-02-11T00:28:16.959 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAVeiRg4i4/ABAAV7sI/umFUqZ1w96g1cmgCQ== 2021-02-11T00:28:16.960 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-11T00:28:17.044 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T00:28:17.127 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T00:28:17.211 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T00:28:17.295 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-11T00:28:17.382 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-11T00:28:17.428 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-02-11T00:28:17.428 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2021-02-11T00:28:17.438 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2021-02-11T00:28:17.536 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2021-02-11T00:28:17.550 DEBUG:teuthology.orchestra.run.smithi057:> rm -- /home/ubuntu/cephtest/c1.monmap 2021-02-11T00:28:17.603 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2021-02-11T00:28:17.603 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2021-02-11T00:28:17.603 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2021-02-11T00:28:17.647 INFO:tasks.ceph.c1.mon.a:Started 2021-02-11T00:28:17.647 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2021-02-11T00:28:17.647 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2021-02-11T00:28:17.647 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2021-02-11T00:28:17.650 INFO:tasks.ceph.c1.mgr.x:Started 2021-02-11T00:28:17.650 INFO:tasks.ceph:Setting crush tunables to default 2021-02-11T00:28:17.650 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster c1 osd crush tunables default 2021-02-11T00:28:18.040 INFO:teuthology.orchestra.run.smithi057.stderr:adjusted tunables profile to default 2021-02-11T00:28:18.054 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2021-02-11T00:28:18.054 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:18.054 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2021-02-11T00:28:18.068 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:18.068 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2021-02-11T00:28:18.123 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:18.124 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2021-02-11T00:28:18.179 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster c1 osd new 0e71d1ee-6011-4ad0-8095-04e77220644e 0 2021-02-11T00:28:18.453 INFO:teuthology.orchestra.run.smithi057.stdout:0 2021-02-11T00:28:18.468 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster c1 osd new eecf95ba-73ea-4fc6-963b-011333d6a634 1 2021-02-11T00:28:18.693 INFO:teuthology.orchestra.run.smithi057.stdout:1 2021-02-11T00:28:18.709 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster c1 osd new 9a0bccf8-a1ae-4ba3-941d-afda1ae4d2a6 2 2021-02-11T00:28:18.928 INFO:teuthology.orchestra.run.smithi057.stdout:2 2021-02-11T00:28:18.942 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2021-02-11T00:28:18.942 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2021-02-11T00:28:18.944 INFO:tasks.ceph.c1.osd.0:Started 2021-02-11T00:28:18.944 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2021-02-11T00:28:18.945 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2021-02-11T00:28:18.947 INFO:tasks.ceph.c1.osd.1:Started 2021-02-11T00:28:18.947 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2021-02-11T00:28:18.947 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2021-02-11T00:28:18.949 INFO:tasks.ceph.c1.osd.2:Started 2021-02-11T00:28:18.949 INFO:tasks.ceph:Waiting for OSDs to come up 2021-02-11T00:28:19.252 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-02-11T00:28:19.295 INFO:tasks.ceph.c1.osd.2.smithi057.stderr:2021-02-11T00:28:19.296+0000 7f2d2b2f2080 -1 Falling back to public interface 2021-02-11T00:28:19.440 INFO:teuthology.misc.health.smithi057.stdout: 2021-02-11T00:28:19.440 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":5,"fsid":"d110339e-4b9e-408b-b214-00afac1a1a23","created":"2021-02-11T00:28:17.792930+0000","modified":"2021-02-11T00:28:18.918982+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"0e71d1ee-6011-4ad0-8095-04e77220644e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eecf95ba-73ea-4fc6-963b-011333d6a634","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9a0bccf8-a1ae-4ba3-941d-afda1ae4d2a6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T00:28:19.452 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-02-11T00:28:19.551 INFO:tasks.ceph.c1.osd.0.smithi057.stderr:2021-02-11T00:28:19.552+0000 7fd7b121a080 -1 Falling back to public interface 2021-02-11T00:28:19.563 INFO:tasks.ceph.c1.osd.1.smithi057.stderr:2021-02-11T00:28:19.564+0000 7f1157e9d080 -1 Falling back to public interface 2021-02-11T00:28:21.989 INFO:tasks.ceph.c1.osd.2.smithi057.stderr:2021-02-11T00:28:21.988+0000 7f2d2b2f2080 -1 osd.2 0 log_to_monitors {default=true} 2021-02-11T00:28:22.535 INFO:tasks.ceph.c1.osd.1.smithi057.stderr:2021-02-11T00:28:22.536+0000 7f1157e9d080 -1 osd.1 0 log_to_monitors {default=true} 2021-02-11T00:28:22.598 INFO:tasks.ceph.c1.osd.0.smithi057.stderr:2021-02-11T00:28:22.600+0000 7fd7b121a080 -1 osd.0 0 log_to_monitors {default=true} 2021-02-11T00:28:23.061 INFO:tasks.ceph.c1.osd.2.smithi057.stderr:2021-02-11T00:28:23.060+0000 7f2d227bb700 -1 osd.2 0 waiting for initial osdmap 2021-02-11T00:28:24.067 INFO:tasks.ceph.c1.osd.1.smithi057.stderr:2021-02-11T00:28:24.068+0000 7f114f366700 -1 osd.1 0 waiting for initial osdmap 2021-02-11T00:28:24.068 INFO:tasks.ceph.c1.osd.0.smithi057.stderr:2021-02-11T00:28:24.068+0000 7fd7a86e3700 -1 osd.0 0 waiting for initial osdmap 2021-02-11T00:28:25.756 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-02-11T00:28:26.055 INFO:teuthology.misc.health.smithi057.stdout: 2021-02-11T00:28:26.055 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":9,"fsid":"d110339e-4b9e-408b-b214-00afac1a1a23","created":"2021-02-11T00:28:17.792930+0000","modified":"2021-02-11T00:28:25.061482+0000","last_up_change":"2021-02-11T00:28:25.061482+0000","last_in_change":"2021-02-11T00:28:25.061482+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"0e71d1ee-6011-4ad0-8095-04e77220644e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":14536},{"type":"v1","addr":"172.21.15.57:6809","nonce":14536}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":14536},{"type":"v1","addr":"172.21.15.57:6811","nonce":14536}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":14536},{"type":"v1","addr":"172.21.15.57:6815","nonce":14536}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":14536},{"type":"v1","addr":"172.21.15.57:6813","nonce":14536}]},"public_addr":"172.21.15.57:6809/14536","cluster_addr":"172.21.15.57:6811/14536","heartbeat_back_addr":"172.21.15.57:6815/14536","heartbeat_front_addr":"172.21.15.57:6813/14536","state":["exists","up"]},{"osd":1,"uuid":"eecf95ba-73ea-4fc6-963b-011333d6a634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":14537},{"type":"v1","addr":"172.21.15.57:6817","nonce":14537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":14537},{"type":"v1","addr":"172.21.15.57:6819","nonce":14537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":14537},{"type":"v1","addr":"172.21.15.57:6823","nonce":14537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":14537},{"type":"v1","addr":"172.21.15.57:6821","nonce":14537}]},"public_addr":"172.21.15.57:6817/14537","cluster_addr":"172.21.15.57:6819/14537","heartbeat_back_addr":"172.21.15.57:6823/14537","heartbeat_front_addr":"172.21.15.57:6821/14537","state":["exists","up"]},{"osd":2,"uuid":"9a0bccf8-a1ae-4ba3-941d-afda1ae4d2a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":14535},{"type":"v1","addr":"172.21.15.57:6801","nonce":14535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":14535},{"type":"v1","addr":"172.21.15.57:6803","nonce":14535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":14535},{"type":"v1","addr":"172.21.15.57:6807","nonce":14535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":14535},{"type":"v1","addr":"172.21.15.57:6805","nonce":14535}]},"public_addr":"172.21.15.57:6801/14535","cluster_addr":"172.21.15.57:6803/14535","heartbeat_back_addr":"172.21.15.57:6807/14535","heartbeat_front_addr":"172.21.15.57:6805/14535","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:22.949429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T00:28:26.068 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-02-11T00:28:26.068 INFO:tasks.ceph:Creating RBD pool 2021-02-11T00:28:26.068 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster c1 osd pool create rbd 8 2021-02-11T00:28:27.101 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'rbd' created 2021-02-11T00:28:27.114 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-02-11T00:28:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:enabled application 'rbd' on pool 'rbd' 2021-02-11T00:28:28.121 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2021-02-11T00:28:28.122 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-02-11T00:28:28.123 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-02-11T00:28:28.417 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:28.417 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":12,"fsid":"d110339e-4b9e-408b-b214-00afac1a1a23","created":"2021-02-11T00:28:17.792930+0000","modified":"2021-02-11T00:28:28.085904+0000","last_up_change":"2021-02-11T00:28:25.061482+0000","last_in_change":"2021-02-11T00:28:25.061482+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T00:28:25.079298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T00:28:26.372533+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0e71d1ee-6011-4ad0-8095-04e77220644e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":14536},{"type":"v1","addr":"172.21.15.57:6809","nonce":14536}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":14536},{"type":"v1","addr":"172.21.15.57:6811","nonce":14536}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":14536},{"type":"v1","addr":"172.21.15.57:6815","nonce":14536}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":14536},{"type":"v1","addr":"172.21.15.57:6813","nonce":14536}]},"public_addr":"172.21.15.57:6809/14536","cluster_addr":"172.21.15.57:6811/14536","heartbeat_back_addr":"172.21.15.57:6815/14536","heartbeat_front_addr":"172.21.15.57:6813/14536","state":["exists","up"]},{"osd":1,"uuid":"eecf95ba-73ea-4fc6-963b-011333d6a634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":14537},{"type":"v1","addr":"172.21.15.57:6817","nonce":14537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":14537},{"type":"v1","addr":"172.21.15.57:6819","nonce":14537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":14537},{"type":"v1","addr":"172.21.15.57:6823","nonce":14537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":14537},{"type":"v1","addr":"172.21.15.57:6821","nonce":14537}]},"public_addr":"172.21.15.57:6817/14537","cluster_addr":"172.21.15.57:6819/14537","heartbeat_back_addr":"172.21.15.57:6823/14537","heartbeat_front_addr":"172.21.15.57:6821/14537","state":["exists","up"]},{"osd":2,"uuid":"9a0bccf8-a1ae-4ba3-941d-afda1ae4d2a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":14535},{"type":"v1","addr":"172.21.15.57:6801","nonce":14535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":14535},{"type":"v1","addr":"172.21.15.57:6803","nonce":14535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":14535},{"type":"v1","addr":"172.21.15.57:6807","nonce":14535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":14535},{"type":"v1","addr":"172.21.15.57:6805","nonce":14535}]},"public_addr":"172.21.15.57:6801/14535","cluster_addr":"172.21.15.57:6803/14535","heartbeat_back_addr":"172.21.15.57:6807/14535","heartbeat_front_addr":"172.21.15.57:6805/14535","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:23.603672+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:23.534457+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:22.949429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T00:28:28.431 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-11T00:28:25.079298+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-02-11T00:28:26.372533+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-11T00:28:28.432 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get device_health_metrics pg_num 2021-02-11T00:28:28.735 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2021-02-11T00:28:28.753 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get rbd pg_num 2021-02-11T00:28:29.055 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 8 2021-02-11T00:28:29.076 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2021-02-11T00:28:29.076 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2021-02-11T00:28:29.076 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2021-02-11T00:28:29.419 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:29.434 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":14351},{"type":"v1","addr":"172.21.15.57:6825","nonce":14351}]},"active_addr":"172.21.15.57:6825/14351","active_change":"2021-02-11T00:28:22.891037+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.57:0","nonce":2028895631}]},{"addrvec":[{"type":"v2","addr":"172.21.15.57:0","nonce":951225374}]}]}} 2021-02-11T00:28:29.436 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2021-02-11T00:28:29.436 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2021-02-11T00:28:29.436 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-02-11T00:28:29.741 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:29.741 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":13,"fsid":"d110339e-4b9e-408b-b214-00afac1a1a23","created":"2021-02-11T00:28:17.792930+0000","modified":"2021-02-11T00:28:29.103154+0000","last_up_change":"2021-02-11T00:28:25.061482+0000","last_in_change":"2021-02-11T00:28:25.061482+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T00:28:25.079298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T00:28:26.372533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0e71d1ee-6011-4ad0-8095-04e77220644e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":14536},{"type":"v1","addr":"172.21.15.57:6809","nonce":14536}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":14536},{"type":"v1","addr":"172.21.15.57:6811","nonce":14536}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":14536},{"type":"v1","addr":"172.21.15.57:6815","nonce":14536}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":14536},{"type":"v1","addr":"172.21.15.57:6813","nonce":14536}]},"public_addr":"172.21.15.57:6809/14536","cluster_addr":"172.21.15.57:6811/14536","heartbeat_back_addr":"172.21.15.57:6815/14536","heartbeat_front_addr":"172.21.15.57:6813/14536","state":["exists","up"]},{"osd":1,"uuid":"eecf95ba-73ea-4fc6-963b-011333d6a634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":14537},{"type":"v1","addr":"172.21.15.57:6817","nonce":14537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":14537},{"type":"v1","addr":"172.21.15.57:6819","nonce":14537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":14537},{"type":"v1","addr":"172.21.15.57:6823","nonce":14537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":14537},{"type":"v1","addr":"172.21.15.57:6821","nonce":14537}]},"public_addr":"172.21.15.57:6817/14537","cluster_addr":"172.21.15.57:6819/14537","heartbeat_back_addr":"172.21.15.57:6823/14537","heartbeat_front_addr":"172.21.15.57:6821/14537","state":["exists","up"]},{"osd":2,"uuid":"9a0bccf8-a1ae-4ba3-941d-afda1ae4d2a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":14535},{"type":"v1","addr":"172.21.15.57:6801","nonce":14535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":14535},{"type":"v1","addr":"172.21.15.57:6803","nonce":14535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":14535},{"type":"v1","addr":"172.21.15.57:6807","nonce":14535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":14535},{"type":"v1","addr":"172.21.15.57:6805","nonce":14535}]},"public_addr":"172.21.15.57:6801/14535","cluster_addr":"172.21.15.57:6803/14535","heartbeat_back_addr":"172.21.15.57:6807/14535","heartbeat_front_addr":"172.21.15.57:6805/14535","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:23.603672+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:23.534457+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:22.949429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T00:28:29.762 INFO:tasks.ceph.ceph_manager.c1:all up! 2021-02-11T00:28:29.762 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-02-11T00:28:30.062 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:30.063 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":13,"fsid":"d110339e-4b9e-408b-b214-00afac1a1a23","created":"2021-02-11T00:28:17.792930+0000","modified":"2021-02-11T00:28:29.103154+0000","last_up_change":"2021-02-11T00:28:25.061482+0000","last_in_change":"2021-02-11T00:28:25.061482+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T00:28:25.079298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T00:28:26.372533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0e71d1ee-6011-4ad0-8095-04e77220644e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":14536},{"type":"v1","addr":"172.21.15.57:6809","nonce":14536}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":14536},{"type":"v1","addr":"172.21.15.57:6811","nonce":14536}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":14536},{"type":"v1","addr":"172.21.15.57:6815","nonce":14536}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":14536},{"type":"v1","addr":"172.21.15.57:6813","nonce":14536}]},"public_addr":"172.21.15.57:6809/14536","cluster_addr":"172.21.15.57:6811/14536","heartbeat_back_addr":"172.21.15.57:6815/14536","heartbeat_front_addr":"172.21.15.57:6813/14536","state":["exists","up"]},{"osd":1,"uuid":"eecf95ba-73ea-4fc6-963b-011333d6a634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":14537},{"type":"v1","addr":"172.21.15.57:6817","nonce":14537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":14537},{"type":"v1","addr":"172.21.15.57:6819","nonce":14537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":14537},{"type":"v1","addr":"172.21.15.57:6823","nonce":14537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":14537},{"type":"v1","addr":"172.21.15.57:6821","nonce":14537}]},"public_addr":"172.21.15.57:6817/14537","cluster_addr":"172.21.15.57:6819/14537","heartbeat_back_addr":"172.21.15.57:6823/14537","heartbeat_front_addr":"172.21.15.57:6821/14537","state":["exists","up"]},{"osd":2,"uuid":"9a0bccf8-a1ae-4ba3-941d-afda1ae4d2a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":14535},{"type":"v1","addr":"172.21.15.57:6801","nonce":14535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":14535},{"type":"v1","addr":"172.21.15.57:6803","nonce":14535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":14535},{"type":"v1","addr":"172.21.15.57:6807","nonce":14535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":14535},{"type":"v1","addr":"172.21.15.57:6805","nonce":14535}]},"public_addr":"172.21.15.57:6801/14535","cluster_addr":"172.21.15.57:6803/14535","heartbeat_back_addr":"172.21.15.57:6807/14535","heartbeat_front_addr":"172.21.15.57:6805/14535","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:23.603672+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:23.534457+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:22.949429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T00:28:30.077 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2021-02-11T00:28:30.251 INFO:teuthology.orchestra.run.smithi057.stdout:38654705668 2021-02-11T00:28:30.252 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2021-02-11T00:28:30.423 INFO:teuthology.orchestra.run.smithi057.stdout:38654705667 2021-02-11T00:28:30.423 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2021-02-11T00:28:30.601 INFO:teuthology.orchestra.run.smithi057.stdout:34359738372 2021-02-11T00:28:30.602 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-02-11T00:28:30.908 INFO:teuthology.orchestra.run.smithi057.stdout:38654705666 2021-02-11T00:28:30.928 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705666 for osd.0 2021-02-11T00:28:31.928 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-02-11T00:28:32.243 INFO:teuthology.orchestra.run.smithi057.stdout:38654705668 2021-02-11T00:28:32.258 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705668 for osd.0 2021-02-11T00:28:32.258 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2021-02-11T00:28:32.561 INFO:teuthology.orchestra.run.smithi057.stdout:38654705667 2021-02-11T00:28:32.576 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705667 for osd.1 2021-02-11T00:28:32.576 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2021-02-11T00:28:32.871 INFO:teuthology.orchestra.run.smithi057.stdout:34359738372 2021-02-11T00:28:32.886 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.2 2021-02-11T00:28:32.886 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2021-02-11T00:28:32.886 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-02-11T00:28:33.190 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:33.192 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2021-02-11T00:28:33.204 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2021-02-11T00:28:32.902727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14680,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":14336,"kb_avail":314558120,"statfs":{"total":322122547200,"available":322107514880,"internally_reserved":0,"allocated":327680,"data_stored":88587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14680064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.122558+0000","last_change":"2021-02-11T00:28:28.119586+0000","last_active":"2021-02-11T00:28:28.122558+0000","last_peered":"2021-02-11T00:28:28.122558+0000","last_clean":"2021-02-11T00:28:28.122558+0000","last_became_active":"2021-02-11T00:28:28.118428+0000","last_became_peered":"2021-02-11T00:28:28.118428+0000","last_unstale":"2021-02-11T00:28:28.122558+0000","last_undegraded":"2021-02-11T00:28:28.122558+0000","last_fullsized":"2021-02-11T00:28:28.122558+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_clean_scrub_stamp":"2021-02-11T00:28:27.079988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.122452+0000","last_change":"2021-02-11T00:28:28.119395+0000","last_active":"2021-02-11T00:28:28.122452+0000","last_peered":"2021-02-11T00:28:28.122452+0000","last_clean":"2021-02-11T00:28:28.122452+0000","last_became_active":"2021-02-11T00:28:28.118479+0000","last_became_peered":"2021-02-11T00:28:28.118479+0000","last_unstale":"2021-02-11T00:28:28.122452+0000","last_undegraded":"2021-02-11T00:28:28.122452+0000","last_fullsized":"2021-02-11T00:28:28.122452+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_clean_scrub_stamp":"2021-02-11T00:28:27.079988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.122490+0000","last_change":"2021-02-11T00:28:28.119948+0000","last_active":"2021-02-11T00:28:28.122490+0000","last_peered":"2021-02-11T00:28:28.122490+0000","last_clean":"2021-02-11T00:28:28.122490+0000","last_became_active":"2021-02-11T00:28:28.118976+0000","last_became_peered":"2021-02-11T00:28:28.118976+0000","last_unstale":"2021-02-11T00:28:28.122490+0000","last_undegraded":"2021-02-11T00:28:28.122490+0000","last_fullsized":"2021-02-11T00:28:28.122490+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_clean_scrub_stamp":"2021-02-11T00:28:27.079988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.121503+0000","last_change":"2021-02-11T00:28:28.118338+0000","last_active":"2021-02-11T00:28:28.121503+0000","last_peered":"2021-02-11T00:28:28.121503+0000","last_clean":"2021-02-11T00:28:28.121503+0000","last_became_active":"2021-02-11T00:28:28.117698+0000","last_became_peered":"2021-02-11T00:28:28.117698+0000","last_unstale":"2021-02-11T00:28:28.121503+0000","last_undegraded":"2021-02-11T00:28:28.121503+0000","last_fullsized":"2021-02-11T00:28:28.121503+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_clean_scrub_stamp":"2021-02-11T00:28:27.079988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.111015+0000","last_change":"2021-02-11T00:28:27.099148+0000","last_active":"2021-02-11T00:28:28.111015+0000","last_peered":"2021-02-11T00:28:28.111015+0000","last_clean":"2021-02-11T00:28:28.111015+0000","last_became_active":"2021-02-11T00:28:27.098669+0000","last_became_peered":"2021-02-11T00:28:27.098669+0000","last_unstale":"2021-02-11T00:28:28.111015+0000","last_undegraded":"2021-02-11T00:28:28.111015+0000","last_fullsized":"2021-02-11T00:28:28.111015+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:26.070965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:26.070965+0000","last_clean_scrub_stamp":"2021-02-11T00:28:26.070965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.119211+0000","last_change":"2021-02-11T00:28:28.113409+0000","last_active":"2021-02-11T00:28:28.119211+0000","last_peered":"2021-02-11T00:28:28.119211+0000","last_clean":"2021-02-11T00:28:28.119211+0000","last_became_active":"2021-02-11T00:28:28.112883+0000","last_became_peered":"2021-02-11T00:28:28.112883+0000","last_unstale":"2021-02-11T00:28:28.119211+0000","last_undegraded":"2021-02-11T00:28:28.119211+0000","last_fullsized":"2021-02-11T00:28:28.119211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_clean_scrub_stamp":"2021-02-11T00:28:27.079988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.121521+0000","last_change":"2021-02-11T00:28:28.118224+0000","last_active":"2021-02-11T00:28:28.121521+0000","last_peered":"2021-02-11T00:28:28.121521+0000","last_clean":"2021-02-11T00:28:28.121521+0000","last_became_active":"2021-02-11T00:28:28.117614+0000","last_became_peered":"2021-02-11T00:28:28.117614+0000","last_unstale":"2021-02-11T00:28:28.121521+0000","last_undegraded":"2021-02-11T00:28:28.121521+0000","last_fullsized":"2021-02-11T00:28:28.121521+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_clean_scrub_stamp":"2021-02-11T00:28:27.079988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.117870+0000","last_change":"2021-02-11T00:28:28.117870+0000","last_active":"2021-02-11T00:28:28.117870+0000","last_peered":"2021-02-11T00:28:28.117870+0000","last_clean":"2021-02-11T00:28:28.117870+0000","last_became_active":"2021-02-11T00:28:28.117387+0000","last_became_peered":"2021-02-11T00:28:28.117387+0000","last_unstale":"2021-02-11T00:28:28.117870+0000","last_undegraded":"2021-02-11T00:28:28.117870+0000","last_fullsized":"2021-02-11T00:28:28.117870+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_clean_scrub_stamp":"2021-02-11T00:28:27.079988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.122424+0000","last_change":"2021-02-11T00:28:28.119482+0000","last_active":"2021-02-11T00:28:28.122424+0000","last_peered":"2021-02-11T00:28:28.122424+0000","last_clean":"2021-02-11T00:28:28.122424+0000","last_became_active":"2021-02-11T00:28:28.118702+0000","last_became_peered":"2021-02-11T00:28:28.118702+0000","last_unstale":"2021-02-11T00:28:28.122424+0000","last_undegraded":"2021-02-11T00:28:28.122424+0000","last_fullsized":"2021-02-11T00:28:28.122424+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_clean_scrub_stamp":"2021-02-11T00:28:27.079988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":31871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5029999999999999}]}]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4848,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":104852752,"statfs":{"total":107374182400,"available":107369218048,"internally_reserved":0,"allocated":106496,"data_stored":28358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.214}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":28358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.665}]}]}]}} 2021-02-11T00:28:33.205 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-02-11T00:28:33.499 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:33.501 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2021-02-11T00:28:33.513 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2021-02-11T00:28:32.902727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14680,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":14336,"kb_avail":314558120,"statfs":{"total":322122547200,"available":322107514880,"internally_reserved":0,"allocated":327680,"data_stored":88587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14680064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.122558+0000","last_change":"2021-02-11T00:28:28.119586+0000","last_active":"2021-02-11T00:28:28.122558+0000","last_peered":"2021-02-11T00:28:28.122558+0000","last_clean":"2021-02-11T00:28:28.122558+0000","last_became_active":"2021-02-11T00:28:28.118428+0000","last_became_peered":"2021-02-11T00:28:28.118428+0000","last_unstale":"2021-02-11T00:28:28.122558+0000","last_undegraded":"2021-02-11T00:28:28.122558+0000","last_fullsized":"2021-02-11T00:28:28.122558+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_clean_scrub_stamp":"2021-02-11T00:28:27.079988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.122452+0000","last_change":"2021-02-11T00:28:28.119395+0000","last_active":"2021-02-11T00:28:28.122452+0000","last_peered":"2021-02-11T00:28:28.122452+0000","last_clean":"2021-02-11T00:28:28.122452+0000","last_became_active":"2021-02-11T00:28:28.118479+0000","last_became_peered":"2021-02-11T00:28:28.118479+0000","last_unstale":"2021-02-11T00:28:28.122452+0000","last_undegraded":"2021-02-11T00:28:28.122452+0000","last_fullsized":"2021-02-11T00:28:28.122452+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_clean_scrub_stamp":"2021-02-11T00:28:27.079988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.122490+0000","last_change":"2021-02-11T00:28:28.119948+0000","last_active":"2021-02-11T00:28:28.122490+0000","last_peered":"2021-02-11T00:28:28.122490+0000","last_clean":"2021-02-11T00:28:28.122490+0000","last_became_active":"2021-02-11T00:28:28.118976+0000","last_became_peered":"2021-02-11T00:28:28.118976+0000","last_unstale":"2021-02-11T00:28:28.122490+0000","last_undegraded":"2021-02-11T00:28:28.122490+0000","last_fullsized":"2021-02-11T00:28:28.122490+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_clean_scrub_stamp":"2021-02-11T00:28:27.079988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.121503+0000","last_change":"2021-02-11T00:28:28.118338+0000","last_active":"2021-02-11T00:28:28.121503+0000","last_peered":"2021-02-11T00:28:28.121503+0000","last_clean":"2021-02-11T00:28:28.121503+0000","last_became_active":"2021-02-11T00:28:28.117698+0000","last_became_peered":"2021-02-11T00:28:28.117698+0000","last_unstale":"2021-02-11T00:28:28.121503+0000","last_undegraded":"2021-02-11T00:28:28.121503+0000","last_fullsized":"2021-02-11T00:28:28.121503+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_clean_scrub_stamp":"2021-02-11T00:28:27.079988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.111015+0000","last_change":"2021-02-11T00:28:27.099148+0000","last_active":"2021-02-11T00:28:28.111015+0000","last_peered":"2021-02-11T00:28:28.111015+0000","last_clean":"2021-02-11T00:28:28.111015+0000","last_became_active":"2021-02-11T00:28:27.098669+0000","last_became_peered":"2021-02-11T00:28:27.098669+0000","last_unstale":"2021-02-11T00:28:28.111015+0000","last_undegraded":"2021-02-11T00:28:28.111015+0000","last_fullsized":"2021-02-11T00:28:28.111015+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:26.070965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:26.070965+0000","last_clean_scrub_stamp":"2021-02-11T00:28:26.070965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.119211+0000","last_change":"2021-02-11T00:28:28.113409+0000","last_active":"2021-02-11T00:28:28.119211+0000","last_peered":"2021-02-11T00:28:28.119211+0000","last_clean":"2021-02-11T00:28:28.119211+0000","last_became_active":"2021-02-11T00:28:28.112883+0000","last_became_peered":"2021-02-11T00:28:28.112883+0000","last_unstale":"2021-02-11T00:28:28.119211+0000","last_undegraded":"2021-02-11T00:28:28.119211+0000","last_fullsized":"2021-02-11T00:28:28.119211+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_clean_scrub_stamp":"2021-02-11T00:28:27.079988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.121521+0000","last_change":"2021-02-11T00:28:28.118224+0000","last_active":"2021-02-11T00:28:28.121521+0000","last_peered":"2021-02-11T00:28:28.121521+0000","last_clean":"2021-02-11T00:28:28.121521+0000","last_became_active":"2021-02-11T00:28:28.117614+0000","last_became_peered":"2021-02-11T00:28:28.117614+0000","last_unstale":"2021-02-11T00:28:28.121521+0000","last_undegraded":"2021-02-11T00:28:28.121521+0000","last_fullsized":"2021-02-11T00:28:28.121521+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_clean_scrub_stamp":"2021-02-11T00:28:27.079988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.117870+0000","last_change":"2021-02-11T00:28:28.117870+0000","last_active":"2021-02-11T00:28:28.117870+0000","last_peered":"2021-02-11T00:28:28.117870+0000","last_clean":"2021-02-11T00:28:28.117870+0000","last_became_active":"2021-02-11T00:28:28.117387+0000","last_became_peered":"2021-02-11T00:28:28.117387+0000","last_unstale":"2021-02-11T00:28:28.117870+0000","last_undegraded":"2021-02-11T00:28:28.117870+0000","last_fullsized":"2021-02-11T00:28:28.117870+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_clean_scrub_stamp":"2021-02-11T00:28:27.079988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:28.122424+0000","last_change":"2021-02-11T00:28:28.119482+0000","last_active":"2021-02-11T00:28:28.122424+0000","last_peered":"2021-02-11T00:28:28.122424+0000","last_clean":"2021-02-11T00:28:28.122424+0000","last_became_active":"2021-02-11T00:28:28.118702+0000","last_became_peered":"2021-02-11T00:28:28.118702+0000","last_unstale":"2021-02-11T00:28:28.122424+0000","last_undegraded":"2021-02-11T00:28:28.122424+0000","last_fullsized":"2021-02-11T00:28:28.122424+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:27.079988+0000","last_clean_scrub_stamp":"2021-02-11T00:28:27.079988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":31871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5029999999999999}]}]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4848,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":104852752,"statfs":{"total":107374182400,"available":107369218048,"internally_reserved":0,"allocated":106496,"data_stored":28358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.214}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":28358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.665}]}]}]}} 2021-02-11T00:28:33.514 INFO:tasks.ceph.ceph_manager.c1:clean! 2021-02-11T00:28:33.514 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2021-02-11T00:28:33.514 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2021-02-11T00:28:33.514 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2021-02-11T00:28:33.872 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:33.893 INFO:teuthology.orchestra.run.smithi057.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-02-11T00:28:33.893 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2021-02-11T00:28:33.893 INFO:teuthology.run_tasks:Running task ceph... 2021-02-11T00:28:33.902 INFO:tasks.ceph:Creating ceph cluster c2... 2021-02-11T00:28:33.902 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-02-11T00:28:33.902 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi/5874431', 'branch': 'wip-datalog-genbacking', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'aemerson@redhat.com', 'first_in_suite': False, 'job_id': '5874431', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-datalog-genbacking'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '597c1051df9f309aa7d6c6db46dcbc2c3982c6e3', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '597c1051df9f309aa7d6c6db46dcbc2c3982c6e3'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-datalog-genbacking', 'sha1': '597c1051df9f309aa7d6c6db46dcbc2c3982c6e3'}}, 'owner': 'scheduled_aemerson@teuthology', 'priority': 68, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '597c1051df9f309aa7d6c6db46dcbc2c3982c6e3', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-datalog-genbacking', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '597c1051df9f309aa7d6c6db46dcbc2c3982c6e3', 'targets': {'smithi057.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKoCvpwFe8apsck2MdrWK3EDZV8EosBNVkB/RSE/yQxHn50Ykjz3HAw8V3iKist9ypyMYwUH6KBS+Nn0x4zE6rbr0psdkO5PoNVvOozmdswDNzLbfFuT3l34PG/m0V/LMMWkTBNdvnxP66WjA8kScX5YLa3TFxCReuQAW8CNPEipAzHVgW3NQhD+M/xR4kwV94O9O/QVEiQsblbbrsas74gf/U58cU8nCWkDiymUUhxl23YvtuxjEfRslH3zoC0L5IMJiW92hCLxccM8HX1ZlO7ZLBiaor4JXRayfSFYwND85bXct1kJ/zjbgIE5ZgnB7FoQHtSZnKeN56W5XOD4Xf', 'smithi095.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj65BMVOnEAISCOlnW3qjVHQt+aIJPq3o/p+le9aqXL2+AQ8VXsBSewIDMWDtIXWmMa//mBFRa9x0oGKXHWPK7w+n23MUBlwQHnTHvPGpczSjnb+PXmwBIZYaTRl7gyl6ts8aecf/FrTM54fhGyMMGYXh3ZIJwuh9NyBMf9BKnRfsBQ6jTkbTKbMKL7XmRisu3EcaelLPM3aiLpuyCYw66OgBdMPvC4Bgvo9+J1lRVdivlw3PuagJn4soMxyAFF6zXJ6DnhAX0PwEsJmK6PrQrBwZ7bFy+lXFq8/bDsGwPrG8cynvH8kMKCjl/VoP1wPKwbg3e8XK/Ci2SQ48VPvMR'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '597c1051df9f309aa7d6c6db46dcbc2c3982c6e3', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '597c1051df9f309aa7d6c6db46dcbc2c3982c6e3', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2021-02-11_00:10:51', 'tube': 'smithi', 'user': 'aemerson', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.29888'} 2021-02-11T00:28:33.903 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-02-11T00:28:33.905 DEBUG:teuthology.orchestra.run.smithi095:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-02-11T00:28:33.912 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m0777 -- /var/run/ceph 2021-02-11T00:28:33.951 DEBUG:teuthology.orchestra.run.smithi095:> sudo install -d -m0777 -- /var/run/ceph 2021-02-11T00:28:33.970 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:33.970 DEBUG:teuthology.orchestra.run.smithi095:> dd if=/scratch_devs of=/dev/stdout 2021-02-11T00:28:34.017 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-11T00:28:34.017 DEBUG:teuthology.orchestra.run.smithi095:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-11T00:28:34.066 INFO:teuthology.orchestra.run.smithi095.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-02-11T00:28:34.066 INFO:teuthology.orchestra.run.smithi095.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T00:28:34.070 INFO:teuthology.orchestra.run.smithi095.stdout:Device: 6h/6d Inode: 343 Links: 1 2021-02-11T00:28:34.070 INFO:teuthology.orchestra.run.smithi095.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T00:28:34.070 INFO:teuthology.orchestra.run.smithi095.stdout:Access: 2021-02-11 00:24:54.233563770 +0000 2021-02-11T00:28:34.070 INFO:teuthology.orchestra.run.smithi095.stdout:Modify: 2021-02-11 00:24:54.157563771 +0000 2021-02-11T00:28:34.070 INFO:teuthology.orchestra.run.smithi095.stdout:Change: 2021-02-11 00:24:54.157563771 +0000 2021-02-11T00:28:34.070 INFO:teuthology.orchestra.run.smithi095.stdout: Birth: - 2021-02-11T00:28:34.076 INFO:teuthology.orchestra.run.smithi095.stderr:1+0 records in 2021-02-11T00:28:34.076 INFO:teuthology.orchestra.run.smithi095.stderr:1+0 records out 2021-02-11T00:28:34.076 INFO:teuthology.orchestra.run.smithi095.stderr:512 bytes copied, 0.000374627 s, 1.4 MB/s 2021-02-11T00:28:34.081 DEBUG:teuthology.orchestra.run.smithi095:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-11T00:28:34.129 INFO:teuthology.orchestra.run.smithi095.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-02-11T00:28:34.130 INFO:teuthology.orchestra.run.smithi095.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T00:28:34.130 INFO:teuthology.orchestra.run.smithi095.stdout:Device: 6h/6d Inode: 349 Links: 1 2021-02-11T00:28:34.130 INFO:teuthology.orchestra.run.smithi095.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T00:28:34.130 INFO:teuthology.orchestra.run.smithi095.stdout:Access: 2021-02-11 00:24:54.461563765 +0000 2021-02-11T00:28:34.130 INFO:teuthology.orchestra.run.smithi095.stdout:Modify: 2021-02-11 00:24:54.357563767 +0000 2021-02-11T00:28:34.130 INFO:teuthology.orchestra.run.smithi095.stdout:Change: 2021-02-11 00:24:54.357563767 +0000 2021-02-11T00:28:34.130 INFO:teuthology.orchestra.run.smithi095.stdout: Birth: - 2021-02-11T00:28:34.139 INFO:teuthology.orchestra.run.smithi095.stderr:1+0 records in 2021-02-11T00:28:34.139 INFO:teuthology.orchestra.run.smithi095.stderr:1+0 records out 2021-02-11T00:28:34.139 INFO:teuthology.orchestra.run.smithi095.stderr:512 bytes copied, 0.00031956 s, 1.6 MB/s 2021-02-11T00:28:34.143 DEBUG:teuthology.orchestra.run.smithi095:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-11T00:28:34.189 INFO:teuthology.orchestra.run.smithi095.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-02-11T00:28:34.190 INFO:teuthology.orchestra.run.smithi095.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T00:28:34.190 INFO:teuthology.orchestra.run.smithi095.stdout:Device: 6h/6d Inode: 355 Links: 1 2021-02-11T00:28:34.190 INFO:teuthology.orchestra.run.smithi095.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T00:28:34.190 INFO:teuthology.orchestra.run.smithi095.stdout:Access: 2021-02-11 00:24:54.709563759 +0000 2021-02-11T00:28:34.190 INFO:teuthology.orchestra.run.smithi095.stdout:Modify: 2021-02-11 00:24:54.605563762 +0000 2021-02-11T00:28:34.190 INFO:teuthology.orchestra.run.smithi095.stdout:Change: 2021-02-11 00:24:54.605563762 +0000 2021-02-11T00:28:34.190 INFO:teuthology.orchestra.run.smithi095.stdout: Birth: - 2021-02-11T00:28:34.199 INFO:teuthology.orchestra.run.smithi095.stderr:1+0 records in 2021-02-11T00:28:34.200 INFO:teuthology.orchestra.run.smithi095.stderr:1+0 records out 2021-02-11T00:28:34.200 INFO:teuthology.orchestra.run.smithi095.stderr:512 bytes copied, 0.000399964 s, 1.3 MB/s 2021-02-11T00:28:34.204 DEBUG:teuthology.orchestra.run.smithi095:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-11T00:28:34.253 INFO:teuthology.orchestra.run.smithi095.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-02-11T00:28:34.253 INFO:teuthology.orchestra.run.smithi095.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-11T00:28:34.253 INFO:teuthology.orchestra.run.smithi095.stdout:Device: 6h/6d Inode: 361 Links: 1 2021-02-11T00:28:34.253 INFO:teuthology.orchestra.run.smithi095.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-11T00:28:34.253 INFO:teuthology.orchestra.run.smithi095.stdout:Access: 2021-02-11 00:24:54.949563754 +0000 2021-02-11T00:28:34.254 INFO:teuthology.orchestra.run.smithi095.stdout:Modify: 2021-02-11 00:24:54.853563756 +0000 2021-02-11T00:28:34.254 INFO:teuthology.orchestra.run.smithi095.stdout:Change: 2021-02-11 00:24:54.853563756 +0000 2021-02-11T00:28:34.254 INFO:teuthology.orchestra.run.smithi095.stdout: Birth: - 2021-02-11T00:28:34.263 INFO:teuthology.orchestra.run.smithi095.stderr:1+0 records in 2021-02-11T00:28:34.263 INFO:teuthology.orchestra.run.smithi095.stderr:1+0 records out 2021-02-11T00:28:34.263 INFO:teuthology.orchestra.run.smithi095.stderr:512 bytes copied, 0.000389468 s, 1.3 MB/s 2021-02-11T00:28:34.267 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T00:28:34.267 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi095.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-11T00:28:34.267 INFO:tasks.ceph:Generating config... 2021-02-11T00:28:34.268 INFO:tasks.ceph:[client] debug rgw = 20 2021-02-11T00:28:34.268 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-02-11T00:28:34.268 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-02-11T00:28:34.268 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T00:28:34.268 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-02-11T00:28:34.269 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-02-11T00:28:34.269 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-02-11T00:28:34.269 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-02-11T00:28:34.269 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-02-11T00:28:34.269 INFO:tasks.ceph:[client] setgroup = ceph 2021-02-11T00:28:34.269 INFO:tasks.ceph:[client] setuser = ceph 2021-02-11T00:28:34.269 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2021-02-11T00:28:34.269 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-02-11T00:28:34.269 INFO:tasks.ceph:[mgr] debug ms = 1 2021-02-11T00:28:34.269 INFO:tasks.ceph:[mon] debug mon = 20 2021-02-11T00:28:34.269 INFO:tasks.ceph:[mon] debug ms = 1 2021-02-11T00:28:34.269 INFO:tasks.ceph:[mon] debug paxos = 20 2021-02-11T00:28:34.269 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2021-02-11T00:28:34.270 INFO:tasks.ceph:[osd] debug filestore = 20 2021-02-11T00:28:34.270 INFO:tasks.ceph:[osd] debug journal = 20 2021-02-11T00:28:34.270 INFO:tasks.ceph:[osd] debug ms = 20 2021-02-11T00:28:34.270 INFO:tasks.ceph:[osd] debug osd = 25 2021-02-11T00:28:34.270 INFO:tasks.ceph:[osd] osd fast shutdown = False 2021-02-11T00:28:34.270 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-02-11T00:28:34.270 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-02-11T00:28:34.270 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-02-11T00:28:34.270 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-02-11T00:28:34.270 INFO:tasks.ceph:Setting up c2.mon.a... 2021-02-11T00:28:34.271 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2021-02-11T00:28:34.339 INFO:teuthology.orchestra.run.smithi095.stdout:creating /etc/ceph/c2.keyring 2021-02-11T00:28:34.341 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2021-02-11T00:28:34.417 DEBUG:teuthology.orchestra.run.smithi095:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-02-11T00:28:34.474 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.95')] 2021-02-11T00:28:34.474 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.95', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-02-11T00:28:34.475 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:34.475 DEBUG:teuthology.orchestra.run.smithi095:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-02-11T00:28:34.521 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.95 --print /home/ubuntu/cephtest/c2.monmap 2021-02-11T00:28:34.596 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:28:34.596 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:28:34.597 INFO:teuthology.orchestra.run.smithi095.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2021-02-11T00:28:34.597 INFO:teuthology.orchestra.run.smithi095.stdout:monmaptool: generated fsid 20d74388-aeac-4108-8e9a-c22b365ad363 2021-02-11T00:28:34.597 INFO:teuthology.orchestra.run.smithi095.stdout:epoch 0 2021-02-11T00:28:34.597 INFO:teuthology.orchestra.run.smithi095.stdout:fsid 20d74388-aeac-4108-8e9a-c22b365ad363 2021-02-11T00:28:34.597 INFO:teuthology.orchestra.run.smithi095.stdout:last_changed 2021-02-11T00:28:34.604922+0000 2021-02-11T00:28:34.597 INFO:teuthology.orchestra.run.smithi095.stdout:created 2021-02-11T00:28:34.604922+0000 2021-02-11T00:28:34.597 INFO:teuthology.orchestra.run.smithi095.stdout:min_mon_release 0 (unknown) 2021-02-11T00:28:34.597 INFO:teuthology.orchestra.run.smithi095.stdout:election_strategy: 1 2021-02-11T00:28:34.597 INFO:teuthology.orchestra.run.smithi095.stdout:0: [v2:172.21.15.95:3300/0,v1:172.21.15.95:6789/0] mon.a 2021-02-11T00:28:34.598 INFO:teuthology.orchestra.run.smithi095.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2021-02-11T00:28:34.598 DEBUG:teuthology.orchestra.run.smithi095:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-02-11T00:28:34.645 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 20d74388-aeac-4108-8e9a-c22b365ad363... 2021-02-11T00:28:34.646 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-02-11T00:28:34.648 DEBUG:teuthology.orchestra.run.smithi095:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-02-11T00:28:34.690 INFO:teuthology.orchestra.run.smithi057.stdout:[global] 2021-02-11T00:28:34.690 INFO:teuthology.orchestra.run.smithi057.stdout: chdir = "" 2021-02-11T00:28:34.690 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-11T00:28:34.690 INFO:teuthology.orchestra.run.smithi057.stdout: auth supported = cephx 2021-02-11T00:28:34.690 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.690 INFO:teuthology.orchestra.run.smithi057.stdout: filestore xattr use omap = true 2021-02-11T00:28:34.691 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.691 INFO:teuthology.orchestra.run.smithi057.stdout: mon clock drift allowed = 1.000 2021-02-11T00:28:34.691 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.691 INFO:teuthology.orchestra.run.smithi057.stdout: osd crush chooseleaf type = 0 2021-02-11T00:28:34.691 INFO:teuthology.orchestra.run.smithi057.stdout: auth debug = true 2021-02-11T00:28:34.691 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.691 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on old message = true 2021-02-11T00:28:34.691 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on bug = true 2021-02-11T00:28:34.691 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.692 INFO:teuthology.orchestra.run.smithi057.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-11T00:28:34.692 INFO:teuthology.orchestra.run.smithi057.stdout: mon pg warn max object skew = 0 2021-02-11T00:28:34.692 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.692 INFO:teuthology.orchestra.run.smithi057.stdout: # disable pg_autoscaler by default for new pools 2021-02-11T00:28:34.692 INFO:teuthology.orchestra.run.smithi057.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-11T00:28:34.692 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.692 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default size = 2 2021-02-11T00:28:34.692 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.692 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow primary affinity = true 2021-02-11T00:28:34.692 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow pg remap = true 2021-02-11T00:28:34.692 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on legacy crush tunables = false 2021-02-11T00:28:34.693 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on crush straw calc version zero = false 2021-02-11T00:28:34.693 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on no sortbitwise = false 2021-02-11T00:28:34.693 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on osd down out interval zero = false 2021-02-11T00:28:34.693 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on too few osds = false 2021-02-11T00:28:34.693 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-11T00:28:34.693 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-11T00:28:34.693 INFO:teuthology.orchestra.run.smithi057.stdout: mon_allow_pool_size_one = true 2021-02-11T00:28:34.693 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.693 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-11T00:28:34.693 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.693 INFO:teuthology.orchestra.run.smithi057.stdout: osd default data pool replay window = 5 2021-02-11T00:28:34.693 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.694 INFO:teuthology.orchestra.run.smithi057.stdout: mon allow pool delete = true 2021-02-11T00:28:34.694 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.694 INFO:teuthology.orchestra.run.smithi057.stdout: mon cluster log file level = debug 2021-02-11T00:28:34.694 INFO:teuthology.orchestra.run.smithi057.stdout: debug asserts on shutdown = true 2021-02-11T00:28:34.694 INFO:teuthology.orchestra.run.smithi057.stdout: mon health detail to clog = false 2021-02-11T00:28:34.694 INFO:teuthology.orchestra.run.smithi057.stdout: mon host = 172.21.15.95 2021-02-11T00:28:34.694 INFO:teuthology.orchestra.run.smithi057.stdout: osd heartbeat grace = 40 2021-02-11T00:28:34.694 INFO:teuthology.orchestra.run.smithi057.stdout: fsid = 20d74388-aeac-4108-8e9a-c22b365ad363 2021-02-11T00:28:34.694 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.694 INFO:teuthology.orchestra.run.smithi057.stdout:[osd] 2021-02-11T00:28:34.694 INFO:teuthology.orchestra.run.smithi057.stdout: osd journal size = 100 2021-02-11T00:28:34.695 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.695 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub load threshold = 5.0 2021-02-11T00:28:34.695 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub max interval = 600 2021-02-11T00:28:34.695 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.695 INFO:teuthology.orchestra.run.smithi057.stdout: osd recover clone overlap = true 2021-02-11T00:28:34.695 INFO:teuthology.orchestra.run.smithi057.stdout: osd recovery max chunk = 1048576 2021-02-11T00:28:34.695 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.695 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug shutdown = true 2021-02-11T00:28:34.695 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug op order = true 2021-02-11T00:28:34.695 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug verify stray on activate = true 2021-02-11T00:28:34.695 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.696 INFO:teuthology.orchestra.run.smithi057.stdout: osd open classes on start = true 2021-02-11T00:28:34.696 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug pg log writeout = true 2021-02-11T00:28:34.696 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.696 INFO:teuthology.orchestra.run.smithi057.stdout: osd deep scrub update digest min age = 30 2021-02-11T00:28:34.696 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.696 INFO:teuthology.orchestra.run.smithi057.stdout: osd map max advance = 10 2021-02-11T00:28:34.696 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.696 INFO:teuthology.orchestra.run.smithi057.stdout: journal zero on create = true 2021-02-11T00:28:34.696 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.696 INFO:teuthology.orchestra.run.smithi057.stdout: filestore ondisk finisher threads = 3 2021-02-11T00:28:34.696 INFO:teuthology.orchestra.run.smithi057.stdout: filestore apply finisher threads = 3 2021-02-11T00:28:34.696 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.697 INFO:teuthology.orchestra.run.smithi057.stdout: bdev debug aio = true 2021-02-11T00:28:34.697 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug misdirected ops = true 2021-02-11T00:28:34.697 INFO:teuthology.orchestra.run.smithi057.stdout: debug filestore = 20 2021-02-11T00:28:34.697 INFO:teuthology.orchestra.run.smithi057.stdout: debug journal = 20 2021-02-11T00:28:34.697 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 20 2021-02-11T00:28:34.697 INFO:teuthology.orchestra.run.smithi057.stdout: debug osd = 25 2021-02-11T00:28:34.697 INFO:teuthology.orchestra.run.smithi057.stdout: osd fast shutdown = False 2021-02-11T00:28:34.697 INFO:teuthology.orchestra.run.smithi057.stdout: rocksdb delete range threshold = 0 2021-02-11T00:28:34.697 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.697 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr] 2021-02-11T00:28:34.697 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2021-02-11T00:28:34.697 INFO:teuthology.orchestra.run.smithi057.stdout: debug mgr = 20 2021-02-11T00:28:34.698 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2021-02-11T00:28:34.698 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2021-02-11T00:28:34.698 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2021-02-11T00:28:34.698 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2021-02-11T00:28:34.698 INFO:teuthology.orchestra.run.smithi057.stdout: mgr/telemetry/nag = false 2021-02-11T00:28:34.698 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.698 INFO:teuthology.orchestra.run.smithi057.stdout:[mon] 2021-02-11T00:28:34.698 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2021-02-11T00:28:34.698 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2021-02-11T00:28:34.699 INFO:teuthology.orchestra.run.smithi057.stdout: debug paxos = 20 2021-02-11T00:28:34.699 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2021-02-11T00:28:34.699 INFO:teuthology.orchestra.run.smithi057.stdout: mon data avail warn = 5 2021-02-11T00:28:34.699 INFO:teuthology.orchestra.run.smithi057.stdout: mon mgr mkfs grace = 240 2021-02-11T00:28:34.699 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2021-02-11T00:28:34.699 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd reporter subtree level = osd 2021-02-11T00:28:34.699 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd prime pg temp = true 2021-02-11T00:28:34.699 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2021-02-11T00:28:34.699 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd crush smoke test = False 2021-02-11T00:28:34.699 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T00:28:34.700 INFO:teuthology.orchestra.run.smithi057.stdout:[client] 2021-02-11T00:28:34.700 INFO:teuthology.orchestra.run.smithi057.stdout: rgw cache enabled = true 2021-02-11T00:28:34.700 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable ops log = true 2021-02-11T00:28:34.700 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable usage log = true 2021-02-11T00:28:34.700 INFO:teuthology.orchestra.run.smithi057.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-11T00:28:34.700 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-11T00:28:34.700 INFO:teuthology.orchestra.run.smithi057.stdout: debug rgw = 20 2021-02-11T00:28:34.700 INFO:teuthology.orchestra.run.smithi057.stdout: rgw crypt require ssl = False 2021-02-11T00:28:34.700 INFO:teuthology.orchestra.run.smithi057.stdout: rgw crypt s3 kms backend = testing 2021-02-11T00:28:34.700 INFO:teuthology.orchestra.run.smithi057.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T00:28:34.701 INFO:teuthology.orchestra.run.smithi057.stdout: rgw curl low speed time = 300 2021-02-11T00:28:34.701 INFO:teuthology.orchestra.run.smithi057.stdout: rgw data log num shards = 4 2021-02-11T00:28:34.701 INFO:teuthology.orchestra.run.smithi057.stdout: rgw md log max shards = 4 2021-02-11T00:28:34.701 INFO:teuthology.orchestra.run.smithi057.stdout: rgw sync log trim interval = 0 2021-02-11T00:28:34.701 INFO:teuthology.orchestra.run.smithi057.stdout: rgw sync obj etag verify = True 2021-02-11T00:28:34.701 INFO:teuthology.orchestra.run.smithi057.stdout: setgroup = ceph 2021-02-11T00:28:34.701 INFO:teuthology.orchestra.run.smithi057.stdout: setuser = ceph 2021-02-11T00:28:34.701 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.a] 2021-02-11T00:28:34.701 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.0] 2021-02-11T00:28:34.701 INFO:teuthology.orchestra.run.smithi057.stdout: osd_max_omap_entries_per_request = 10 2021-02-11T00:28:34.702 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.1] 2021-02-11T00:28:34.702 INFO:teuthology.orchestra.run.smithi057.stdout: osd_max_omap_entries_per_request = 1000 2021-02-11T00:28:34.702 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.2] 2021-02-11T00:28:34.702 INFO:teuthology.orchestra.run.smithi057.stdout: osd_max_omap_entries_per_request = 10000 2021-02-11T00:28:34.718 INFO:teuthology.orchestra.run.smithi095.stdout:[global] 2021-02-11T00:28:34.718 INFO:teuthology.orchestra.run.smithi095.stdout: chdir = "" 2021-02-11T00:28:34.718 INFO:teuthology.orchestra.run.smithi095.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-11T00:28:34.718 INFO:teuthology.orchestra.run.smithi095.stdout: auth supported = cephx 2021-02-11T00:28:34.719 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.719 INFO:teuthology.orchestra.run.smithi095.stdout: filestore xattr use omap = true 2021-02-11T00:28:34.719 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.719 INFO:teuthology.orchestra.run.smithi095.stdout: mon clock drift allowed = 1.000 2021-02-11T00:28:34.719 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.719 INFO:teuthology.orchestra.run.smithi095.stdout: osd crush chooseleaf type = 0 2021-02-11T00:28:34.719 INFO:teuthology.orchestra.run.smithi095.stdout: auth debug = true 2021-02-11T00:28:34.719 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.719 INFO:teuthology.orchestra.run.smithi095.stdout: ms die on old message = true 2021-02-11T00:28:34.719 INFO:teuthology.orchestra.run.smithi095.stdout: ms die on bug = true 2021-02-11T00:28:34.719 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.720 INFO:teuthology.orchestra.run.smithi095.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-11T00:28:34.720 INFO:teuthology.orchestra.run.smithi095.stdout: mon pg warn max object skew = 0 2021-02-11T00:28:34.720 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.720 INFO:teuthology.orchestra.run.smithi095.stdout: # disable pg_autoscaler by default for new pools 2021-02-11T00:28:34.720 INFO:teuthology.orchestra.run.smithi095.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-11T00:28:34.720 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.720 INFO:teuthology.orchestra.run.smithi095.stdout: osd pool default size = 2 2021-02-11T00:28:34.720 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.720 INFO:teuthology.orchestra.run.smithi095.stdout: mon osd allow primary affinity = true 2021-02-11T00:28:34.720 INFO:teuthology.orchestra.run.smithi095.stdout: mon osd allow pg remap = true 2021-02-11T00:28:34.721 INFO:teuthology.orchestra.run.smithi095.stdout: mon warn on legacy crush tunables = false 2021-02-11T00:28:34.721 INFO:teuthology.orchestra.run.smithi095.stdout: mon warn on crush straw calc version zero = false 2021-02-11T00:28:34.721 INFO:teuthology.orchestra.run.smithi095.stdout: mon warn on no sortbitwise = false 2021-02-11T00:28:34.721 INFO:teuthology.orchestra.run.smithi095.stdout: mon warn on osd down out interval zero = false 2021-02-11T00:28:34.721 INFO:teuthology.orchestra.run.smithi095.stdout: mon warn on too few osds = false 2021-02-11T00:28:34.721 INFO:teuthology.orchestra.run.smithi095.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-11T00:28:34.721 INFO:teuthology.orchestra.run.smithi095.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-11T00:28:34.721 INFO:teuthology.orchestra.run.smithi095.stdout: mon_allow_pool_size_one = true 2021-02-11T00:28:34.721 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.721 INFO:teuthology.orchestra.run.smithi095.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-11T00:28:34.721 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.721 INFO:teuthology.orchestra.run.smithi095.stdout: osd default data pool replay window = 5 2021-02-11T00:28:34.722 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.722 INFO:teuthology.orchestra.run.smithi095.stdout: mon allow pool delete = true 2021-02-11T00:28:34.722 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.722 INFO:teuthology.orchestra.run.smithi095.stdout: mon cluster log file level = debug 2021-02-11T00:28:34.722 INFO:teuthology.orchestra.run.smithi095.stdout: debug asserts on shutdown = true 2021-02-11T00:28:34.722 INFO:teuthology.orchestra.run.smithi095.stdout: mon health detail to clog = false 2021-02-11T00:28:34.722 INFO:teuthology.orchestra.run.smithi095.stdout: mon host = 172.21.15.95 2021-02-11T00:28:34.722 INFO:teuthology.orchestra.run.smithi095.stdout: osd heartbeat grace = 40 2021-02-11T00:28:34.722 INFO:teuthology.orchestra.run.smithi095.stdout: fsid = 20d74388-aeac-4108-8e9a-c22b365ad363 2021-02-11T00:28:34.722 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.723 INFO:teuthology.orchestra.run.smithi095.stdout:[osd] 2021-02-11T00:28:34.723 INFO:teuthology.orchestra.run.smithi095.stdout: osd journal size = 100 2021-02-11T00:28:34.723 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.723 INFO:teuthology.orchestra.run.smithi095.stdout: osd scrub load threshold = 5.0 2021-02-11T00:28:34.723 INFO:teuthology.orchestra.run.smithi095.stdout: osd scrub max interval = 600 2021-02-11T00:28:34.723 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.723 INFO:teuthology.orchestra.run.smithi095.stdout: osd recover clone overlap = true 2021-02-11T00:28:34.723 INFO:teuthology.orchestra.run.smithi095.stdout: osd recovery max chunk = 1048576 2021-02-11T00:28:34.723 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.724 INFO:teuthology.orchestra.run.smithi095.stdout: osd debug shutdown = true 2021-02-11T00:28:34.724 INFO:teuthology.orchestra.run.smithi095.stdout: osd debug op order = true 2021-02-11T00:28:34.724 INFO:teuthology.orchestra.run.smithi095.stdout: osd debug verify stray on activate = true 2021-02-11T00:28:34.724 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.724 INFO:teuthology.orchestra.run.smithi095.stdout: osd open classes on start = true 2021-02-11T00:28:34.724 INFO:teuthology.orchestra.run.smithi095.stdout: osd debug pg log writeout = true 2021-02-11T00:28:34.724 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.724 INFO:teuthology.orchestra.run.smithi095.stdout: osd deep scrub update digest min age = 30 2021-02-11T00:28:34.724 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.724 INFO:teuthology.orchestra.run.smithi095.stdout: osd map max advance = 10 2021-02-11T00:28:34.724 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.725 INFO:teuthology.orchestra.run.smithi095.stdout: journal zero on create = true 2021-02-11T00:28:34.725 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.725 INFO:teuthology.orchestra.run.smithi095.stdout: filestore ondisk finisher threads = 3 2021-02-11T00:28:34.725 INFO:teuthology.orchestra.run.smithi095.stdout: filestore apply finisher threads = 3 2021-02-11T00:28:34.725 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.725 INFO:teuthology.orchestra.run.smithi095.stdout: bdev debug aio = true 2021-02-11T00:28:34.725 INFO:teuthology.orchestra.run.smithi095.stdout: osd debug misdirected ops = true 2021-02-11T00:28:34.725 INFO:teuthology.orchestra.run.smithi095.stdout: debug filestore = 20 2021-02-11T00:28:34.726 INFO:teuthology.orchestra.run.smithi095.stdout: debug journal = 20 2021-02-11T00:28:34.726 INFO:teuthology.orchestra.run.smithi095.stdout: debug ms = 20 2021-02-11T00:28:34.726 INFO:teuthology.orchestra.run.smithi095.stdout: debug osd = 25 2021-02-11T00:28:34.726 INFO:teuthology.orchestra.run.smithi095.stdout: osd fast shutdown = False 2021-02-11T00:28:34.726 INFO:teuthology.orchestra.run.smithi095.stdout: rocksdb delete range threshold = 0 2021-02-11T00:28:34.726 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.726 INFO:teuthology.orchestra.run.smithi095.stdout:[mgr] 2021-02-11T00:28:34.726 INFO:teuthology.orchestra.run.smithi095.stdout: debug ms = 1 2021-02-11T00:28:34.726 INFO:teuthology.orchestra.run.smithi095.stdout: debug mgr = 20 2021-02-11T00:28:34.727 INFO:teuthology.orchestra.run.smithi095.stdout: debug mon = 20 2021-02-11T00:28:34.727 INFO:teuthology.orchestra.run.smithi095.stdout: debug auth = 20 2021-02-11T00:28:34.727 INFO:teuthology.orchestra.run.smithi095.stdout: mon reweight min pgs per osd = 4 2021-02-11T00:28:34.727 INFO:teuthology.orchestra.run.smithi095.stdout: mon reweight min bytes per osd = 10 2021-02-11T00:28:34.727 INFO:teuthology.orchestra.run.smithi095.stdout: mgr/telemetry/nag = false 2021-02-11T00:28:34.727 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.727 INFO:teuthology.orchestra.run.smithi095.stdout:[mon] 2021-02-11T00:28:34.727 INFO:teuthology.orchestra.run.smithi095.stdout: debug ms = 1 2021-02-11T00:28:34.727 INFO:teuthology.orchestra.run.smithi095.stdout: debug mon = 20 2021-02-11T00:28:34.727 INFO:teuthology.orchestra.run.smithi095.stdout: debug paxos = 20 2021-02-11T00:28:34.728 INFO:teuthology.orchestra.run.smithi095.stdout: debug auth = 20 2021-02-11T00:28:34.728 INFO:teuthology.orchestra.run.smithi095.stdout: mon data avail warn = 5 2021-02-11T00:28:34.728 INFO:teuthology.orchestra.run.smithi095.stdout: mon mgr mkfs grace = 240 2021-02-11T00:28:34.728 INFO:teuthology.orchestra.run.smithi095.stdout: mon reweight min pgs per osd = 4 2021-02-11T00:28:34.728 INFO:teuthology.orchestra.run.smithi095.stdout: mon osd reporter subtree level = osd 2021-02-11T00:28:34.728 INFO:teuthology.orchestra.run.smithi095.stdout: mon osd prime pg temp = true 2021-02-11T00:28:34.728 INFO:teuthology.orchestra.run.smithi095.stdout: mon reweight min bytes per osd = 10 2021-02-11T00:28:34.728 INFO:teuthology.orchestra.run.smithi095.stdout: mon osd crush smoke test = False 2021-02-11T00:28:34.728 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:34.729 INFO:teuthology.orchestra.run.smithi095.stdout:[client] 2021-02-11T00:28:34.729 INFO:teuthology.orchestra.run.smithi095.stdout: rgw cache enabled = true 2021-02-11T00:28:34.729 INFO:teuthology.orchestra.run.smithi095.stdout: rgw enable ops log = true 2021-02-11T00:28:34.729 INFO:teuthology.orchestra.run.smithi095.stdout: rgw enable usage log = true 2021-02-11T00:28:34.729 INFO:teuthology.orchestra.run.smithi095.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-11T00:28:34.729 INFO:teuthology.orchestra.run.smithi095.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-11T00:28:34.729 INFO:teuthology.orchestra.run.smithi095.stdout: debug rgw = 20 2021-02-11T00:28:34.730 INFO:teuthology.orchestra.run.smithi095.stdout: rgw crypt require ssl = False 2021-02-11T00:28:34.730 INFO:teuthology.orchestra.run.smithi095.stdout: rgw crypt s3 kms backend = testing 2021-02-11T00:28:34.730 INFO:teuthology.orchestra.run.smithi095.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-02-11T00:28:34.730 INFO:teuthology.orchestra.run.smithi095.stdout: rgw curl low speed time = 300 2021-02-11T00:28:34.730 INFO:teuthology.orchestra.run.smithi095.stdout: rgw data log num shards = 4 2021-02-11T00:28:34.730 INFO:teuthology.orchestra.run.smithi095.stdout: rgw md log max shards = 4 2021-02-11T00:28:34.730 INFO:teuthology.orchestra.run.smithi095.stdout: rgw sync log trim interval = 0 2021-02-11T00:28:34.730 INFO:teuthology.orchestra.run.smithi095.stdout: rgw sync obj etag verify = True 2021-02-11T00:28:34.730 INFO:teuthology.orchestra.run.smithi095.stdout: setgroup = ceph 2021-02-11T00:28:34.730 INFO:teuthology.orchestra.run.smithi095.stdout: setuser = ceph 2021-02-11T00:28:34.731 INFO:teuthology.orchestra.run.smithi095.stdout:[mon.a] 2021-02-11T00:28:34.731 INFO:teuthology.orchestra.run.smithi095.stdout:[osd.0] 2021-02-11T00:28:34.731 INFO:teuthology.orchestra.run.smithi095.stdout: osd_max_omap_entries_per_request = 10 2021-02-11T00:28:34.731 INFO:teuthology.orchestra.run.smithi095.stdout:[osd.1] 2021-02-11T00:28:34.731 INFO:teuthology.orchestra.run.smithi095.stdout: osd_max_omap_entries_per_request = 1000 2021-02-11T00:28:34.731 INFO:teuthology.orchestra.run.smithi095.stdout:[osd.2] 2021-02-11T00:28:34.731 INFO:teuthology.orchestra.run.smithi095.stdout: osd_max_omap_entries_per_request = 10000 2021-02-11T00:28:34.732 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2021-02-11T00:28:34.732 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2021-02-11T00:28:34.821 INFO:tasks.ceph:Copying monmap to all nodes... 2021-02-11T00:28:34.821 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:34.821 DEBUG:teuthology.orchestra.run.smithi095:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2021-02-11T00:28:34.825 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:34.825 DEBUG:teuthology.orchestra.run.smithi095:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2021-02-11T00:28:34.873 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi057.front.sepia.ceph.com 2021-02-11T00:28:34.873 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:34.873 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/c2.keyring 2021-02-11T00:28:34.873 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-02-11T00:28:34.897 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2021-02-11T00:28:34.897 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-02-11T00:28:34.947 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi095.front.sepia.ceph.com 2021-02-11T00:28:34.947 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:34.947 DEBUG:teuthology.orchestra.run.smithi095:> sudo dd of=/etc/ceph/c2.keyring 2021-02-11T00:28:34.947 DEBUG:teuthology.orchestra.run.smithi095:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-02-11T00:28:34.981 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:34.981 DEBUG:teuthology.orchestra.run.smithi095:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-02-11T00:28:35.029 INFO:tasks.ceph:Setting up mon nodes... 2021-02-11T00:28:35.029 INFO:tasks.ceph:Setting up mgr nodes... 2021-02-11T00:28:35.030 DEBUG:teuthology.orchestra.run.smithi095:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2021-02-11T00:28:35.115 INFO:teuthology.orchestra.run.smithi095.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2021-02-11T00:28:35.117 INFO:tasks.ceph:Setting up mds nodes... 2021-02-11T00:28:35.118 INFO:tasks.ceph_client:Setting up client nodes... 2021-02-11T00:28:35.118 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2021-02-11T00:28:35.192 INFO:teuthology.orchestra.run.smithi095.stdout:creating /etc/ceph/c2.client.0.keyring 2021-02-11T00:28:35.203 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2021-02-11T00:28:35.279 INFO:teuthology.orchestra.run.smithi095.stdout:creating /etc/ceph/c2.client.1.keyring 2021-02-11T00:28:35.291 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-02-11T00:28:35.291 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi057.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi095.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-11T00:28:35.291 DEBUG:teuthology.orchestra.run.smithi095:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2021-02-11T00:28:35.348 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T00:28:35.348 INFO:tasks.ceph:role: c2.osd.0 2021-02-11T00:28:35.348 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi095.front.sepia.ceph.com 2021-02-11T00:28:35.348 DEBUG:teuthology.orchestra.run.smithi095:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-02-11T00:28:35.869 INFO:teuthology.orchestra.run.smithi095.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T00:28:35.870 INFO:teuthology.orchestra.run.smithi095.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T00:28:35.870 INFO:teuthology.orchestra.run.smithi095.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T00:28:35.870 INFO:teuthology.orchestra.run.smithi095.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T00:28:35.870 INFO:teuthology.orchestra.run.smithi095.stdout: = sunit=0 swidth=0 blks 2021-02-11T00:28:35.870 INFO:teuthology.orchestra.run.smithi095.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T00:28:35.870 INFO:teuthology.orchestra.run.smithi095.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T00:28:35.870 INFO:teuthology.orchestra.run.smithi095.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T00:28:35.870 INFO:teuthology.orchestra.run.smithi095.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T00:28:35.871 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi095.front.sepia.ceph.com -o noatime 2021-02-11T00:28:35.871 DEBUG:teuthology.orchestra.run.smithi095:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2021-02-11T00:28:35.951 DEBUG:teuthology.orchestra.run.smithi095:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2021-02-11T00:28:35.962 INFO:teuthology.orchestra.run.smithi095.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T00:28:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T00:28:35.963 DEBUG:teuthology.orchestra.run.smithi095:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2021-02-11T00:28:36.022 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T00:28:36.022 INFO:tasks.ceph:role: c2.osd.1 2021-02-11T00:28:36.022 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi095.front.sepia.ceph.com 2021-02-11T00:28:36.022 DEBUG:teuthology.orchestra.run.smithi095:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-02-11T00:28:36.563 INFO:teuthology.orchestra.run.smithi095.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T00:28:36.563 INFO:teuthology.orchestra.run.smithi095.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T00:28:36.563 INFO:teuthology.orchestra.run.smithi095.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T00:28:36.563 INFO:teuthology.orchestra.run.smithi095.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T00:28:36.563 INFO:teuthology.orchestra.run.smithi095.stdout: = sunit=0 swidth=0 blks 2021-02-11T00:28:36.563 INFO:teuthology.orchestra.run.smithi095.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T00:28:36.563 INFO:teuthology.orchestra.run.smithi095.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T00:28:36.563 INFO:teuthology.orchestra.run.smithi095.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T00:28:36.563 INFO:teuthology.orchestra.run.smithi095.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T00:28:36.565 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi095.front.sepia.ceph.com -o noatime 2021-02-11T00:28:36.565 DEBUG:teuthology.orchestra.run.smithi095:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2021-02-11T00:28:36.620 DEBUG:teuthology.orchestra.run.smithi095:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2021-02-11T00:28:36.631 INFO:teuthology.orchestra.run.smithi095.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T00:28:36.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T00:28:36.632 DEBUG:teuthology.orchestra.run.smithi095:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2021-02-11T00:28:36.688 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-02-11T00:28:36.688 INFO:tasks.ceph:role: c2.osd.2 2021-02-11T00:28:36.688 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi095.front.sepia.ceph.com 2021-02-11T00:28:36.688 DEBUG:teuthology.orchestra.run.smithi095:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-02-11T00:28:37.200 INFO:teuthology.orchestra.run.smithi095.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-02-11T00:28:37.200 INFO:teuthology.orchestra.run.smithi095.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-11T00:28:37.200 INFO:teuthology.orchestra.run.smithi095.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-02-11T00:28:37.200 INFO:teuthology.orchestra.run.smithi095.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-11T00:28:37.200 INFO:teuthology.orchestra.run.smithi095.stdout: = sunit=0 swidth=0 blks 2021-02-11T00:28:37.201 INFO:teuthology.orchestra.run.smithi095.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-11T00:28:37.201 INFO:teuthology.orchestra.run.smithi095.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-11T00:28:37.201 INFO:teuthology.orchestra.run.smithi095.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-11T00:28:37.201 INFO:teuthology.orchestra.run.smithi095.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-11T00:28:37.202 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi095.front.sepia.ceph.com -o noatime 2021-02-11T00:28:37.202 DEBUG:teuthology.orchestra.run.smithi095:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2021-02-11T00:28:37.249 DEBUG:teuthology.orchestra.run.smithi095:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2021-02-11T00:28:37.301 INFO:teuthology.orchestra.run.smithi095.stderr:sudo: /sbin/restorecon: command not found 2021-02-11T00:28:37.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T00:28:37.302 DEBUG:teuthology.orchestra.run.smithi095:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2021-02-11T00:28:37.386 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:37.388+0000 7f84e9f9d080 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2021-02-11T00:28:37.387 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:37.388+0000 7f84e9f9d080 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2021-02-11T00:28:37.387 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:37.388+0000 7f84e9f9d080 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-02-11T00:28:37.387 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:37.388+0000 7f84e9f9d080 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-02-11T00:28:37.387 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:37.388+0000 7f84e9f9d080 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-02-11T00:28:37.390 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:37.392+0000 7f84e9f9d080 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2021-02-11T00:28:39.556 DEBUG:teuthology.orchestra.run.smithi095:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2021-02-11T00:28:41.060 DEBUG:teuthology.orchestra.run.smithi095:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2021-02-11T00:28:41.142 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:41.144+0000 7f0f2a973080 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2021-02-11T00:28:41.142 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:41.144+0000 7f0f2a973080 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2021-02-11T00:28:41.142 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:41.144+0000 7f0f2a973080 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-02-11T00:28:41.142 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:41.144+0000 7f0f2a973080 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-02-11T00:28:41.142 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:41.144+0000 7f0f2a973080 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-02-11T00:28:41.143 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:41.144+0000 7f0f2a973080 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2021-02-11T00:28:43.327 DEBUG:teuthology.orchestra.run.smithi095:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2021-02-11T00:28:43.386 DEBUG:teuthology.orchestra.run.smithi095:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2021-02-11T00:28:43.466 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:43.468+0000 7f683a1c3080 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2021-02-11T00:28:43.466 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:43.468+0000 7f683a1c3080 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2021-02-11T00:28:43.466 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:43.468+0000 7f683a1c3080 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-02-11T00:28:43.466 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:43.468+0000 7f683a1c3080 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-02-11T00:28:43.466 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:43.468+0000 7f683a1c3080 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-02-11T00:28:43.466 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:28:43.468+0000 7f683a1c3080 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2021-02-11T00:28:45.631 DEBUG:teuthology.orchestra.run.smithi095:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2021-02-11T00:28:45.686 INFO:tasks.ceph:Reading keys from all nodes... 2021-02-11T00:28:45.686 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:45.687 DEBUG:teuthology.orchestra.run.smithi095:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2021-02-11T00:28:45.741 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:45.741 DEBUG:teuthology.orchestra.run.smithi095:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2021-02-11T00:28:45.796 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:45.797 DEBUG:teuthology.orchestra.run.smithi095:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2021-02-11T00:28:45.854 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:45.854 DEBUG:teuthology.orchestra.run.smithi095:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2021-02-11T00:28:45.909 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:45.909 DEBUG:teuthology.orchestra.run.smithi095:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2021-02-11T00:28:45.959 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:45.959 DEBUG:teuthology.orchestra.run.smithi095:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2021-02-11T00:28:46.009 INFO:tasks.ceph:Adding keys to all mons... 2021-02-11T00:28:46.009 DEBUG:teuthology.orchestra.run.smithi095:> sudo tee -a /etc/ceph/c2.keyring 2021-02-11T00:28:46.064 INFO:teuthology.orchestra.run.smithi095.stdout:[mgr.x] 2021-02-11T00:28:46.064 INFO:teuthology.orchestra.run.smithi095.stdout: key = AQAzeiRgUBFeBxAA+68Y0ZXJ22XsQpigoFhCtA== 2021-02-11T00:28:46.064 INFO:teuthology.orchestra.run.smithi095.stdout:[osd.0] 2021-02-11T00:28:46.064 INFO:teuthology.orchestra.run.smithi095.stdout: key = AQA1eiRgMo1+FxAAqdbyCxm3er7nsElt7nJdqg== 2021-02-11T00:28:46.065 INFO:teuthology.orchestra.run.smithi095.stdout:[osd.1] 2021-02-11T00:28:46.065 INFO:teuthology.orchestra.run.smithi095.stdout: key = AQA5eiRgG9XxCBAAz5+cC3EZm3HpPdZekstuJw== 2021-02-11T00:28:46.065 INFO:teuthology.orchestra.run.smithi095.stdout:[osd.2] 2021-02-11T00:28:46.065 INFO:teuthology.orchestra.run.smithi095.stdout: key = AQA7eiRgYig9HBAAtYJrVDf9KYQn6KsOCHUhnA== 2021-02-11T00:28:46.065 INFO:teuthology.orchestra.run.smithi095.stdout:[client.0] 2021-02-11T00:28:46.065 INFO:teuthology.orchestra.run.smithi095.stdout: key = AQAzeiRgwjH2CxAAYTEgpCwEsaLyv5oOTIAG+w== 2021-02-11T00:28:46.065 INFO:teuthology.orchestra.run.smithi095.stdout:[client.1] 2021-02-11T00:28:46.065 INFO:teuthology.orchestra.run.smithi095.stdout: key = AQAzeiRg9zIrERAAicIBuw4gDSJi0KtuIhprfg== 2021-02-11T00:28:46.066 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-11T00:28:46.149 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T00:28:46.233 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T00:28:46.316 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-11T00:28:46.401 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-11T00:28:46.491 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-11T00:28:46.575 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-02-11T00:28:46.575 DEBUG:teuthology.orchestra.run.smithi095:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2021-02-11T00:28:46.632 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2021-02-11T00:28:46.730 DEBUG:teuthology.orchestra.run.smithi095:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2021-02-11T00:28:46.745 DEBUG:teuthology.orchestra.run.smithi095:> rm -- /home/ubuntu/cephtest/c2.monmap 2021-02-11T00:28:46.793 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2021-02-11T00:28:46.793 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2021-02-11T00:28:46.793 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2021-02-11T00:28:46.837 INFO:tasks.ceph.c2.mon.a:Started 2021-02-11T00:28:46.837 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2021-02-11T00:28:46.837 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2021-02-11T00:28:46.837 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2021-02-11T00:28:46.840 INFO:tasks.ceph.c2.mgr.x:Started 2021-02-11T00:28:46.840 INFO:tasks.ceph:Setting crush tunables to default 2021-02-11T00:28:46.840 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph --cluster c2 osd crush tunables default 2021-02-11T00:28:47.240 INFO:teuthology.orchestra.run.smithi095.stderr:adjusted tunables profile to default 2021-02-11T00:28:47.251 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2021-02-11T00:28:47.252 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:47.252 DEBUG:teuthology.orchestra.run.smithi095:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2021-02-11T00:28:47.264 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:47.264 DEBUG:teuthology.orchestra.run.smithi095:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2021-02-11T00:28:47.321 DEBUG:teuthology.orchestra.run.smithi095:> set -ex 2021-02-11T00:28:47.321 DEBUG:teuthology.orchestra.run.smithi095:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2021-02-11T00:28:47.377 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph --cluster c2 osd new e3cce55d-c182-4525-a567-52c86d5dc3db 0 2021-02-11T00:28:47.643 INFO:teuthology.orchestra.run.smithi095.stdout:0 2021-02-11T00:28:47.659 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph --cluster c2 osd new 618c16cc-ae3f-4158-8f89-2c2e7976780c 1 2021-02-11T00:28:47.869 INFO:teuthology.orchestra.run.smithi095.stdout:1 2021-02-11T00:28:47.882 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph --cluster c2 osd new 91671609-8e48-43aa-9b1f-f64b54a33d9a 2 2021-02-11T00:28:48.098 INFO:teuthology.orchestra.run.smithi095.stdout:2 2021-02-11T00:28:48.117 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2021-02-11T00:28:48.117 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2021-02-11T00:28:48.120 INFO:tasks.ceph.c2.osd.0:Started 2021-02-11T00:28:48.120 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2021-02-11T00:28:48.120 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2021-02-11T00:28:48.122 INFO:tasks.ceph.c2.osd.1:Started 2021-02-11T00:28:48.122 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2021-02-11T00:28:48.122 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2021-02-11T00:28:48.124 INFO:tasks.ceph.c2.osd.2:Started 2021-02-11T00:28:48.124 INFO:tasks.ceph:Waiting for OSDs to come up 2021-02-11T00:28:48.427 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-02-11T00:28:48.493 INFO:tasks.ceph.c2.osd.1.smithi095.stderr:2021-02-11T00:28:48.496+0000 7f66f4274080 -1 Falling back to public interface 2021-02-11T00:28:48.613 INFO:teuthology.misc.health.smithi095.stdout: 2021-02-11T00:28:48.614 INFO:teuthology.misc.health.smithi095.stdout:{"epoch":5,"fsid":"20d74388-aeac-4108-8e9a-c22b365ad363","created":"2021-02-11T00:28:46.956056+0000","modified":"2021-02-11T00:28:48.103173+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"e3cce55d-c182-4525-a567-52c86d5dc3db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"618c16cc-ae3f-4158-8f89-2c2e7976780c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"91671609-8e48-43aa-9b1f-f64b54a33d9a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T00:28:48.625 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-02-11T00:28:48.726 INFO:tasks.ceph.c2.osd.0.smithi095.stderr:2021-02-11T00:28:48.728+0000 7faea8f92080 -1 Falling back to public interface 2021-02-11T00:28:48.727 INFO:tasks.ceph.c2.osd.2.smithi095.stderr:2021-02-11T00:28:48.728+0000 7f12d53a4080 -1 Falling back to public interface 2021-02-11T00:28:51.460 INFO:tasks.ceph.c2.osd.1.smithi095.stderr:2021-02-11T00:28:51.463+0000 7f66f4274080 -1 osd.1 0 log_to_monitors {default=true} 2021-02-11T00:28:51.680 INFO:tasks.ceph.c2.osd.0.smithi095.stderr:2021-02-11T00:28:51.687+0000 7faea8f92080 -1 osd.0 0 log_to_monitors {default=true} 2021-02-11T00:28:51.745 INFO:tasks.ceph.c2.osd.2.smithi095.stderr:2021-02-11T00:28:51.751+0000 7f12d53a4080 -1 osd.2 0 log_to_monitors {default=true} 2021-02-11T00:28:52.533 INFO:tasks.ceph.c2.osd.1.smithi095.stderr:2021-02-11T00:28:52.539+0000 7f66eb73d700 -1 osd.1 0 waiting for initial osdmap 2021-02-11T00:28:53.532 INFO:tasks.ceph.c2.osd.0.smithi095.stderr:2021-02-11T00:28:53.534+0000 7faea1c5e700 -1 osd.0 0 waiting for initial osdmap 2021-02-11T00:28:53.532 INFO:tasks.ceph.c2.osd.2.smithi095.stderr:2021-02-11T00:28:53.534+0000 7f12cc86d700 -1 osd.2 0 waiting for initial osdmap 2021-02-11T00:28:54.929 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-02-11T00:28:55.231 INFO:teuthology.misc.health.smithi095.stdout: 2021-02-11T00:28:55.232 INFO:teuthology.misc.health.smithi095.stdout:{"epoch":9,"fsid":"20d74388-aeac-4108-8e9a-c22b365ad363","created":"2021-02-11T00:28:46.956056+0000","modified":"2021-02-11T00:28:54.536787+0000","last_up_change":"2021-02-11T00:28:54.536787+0000","last_in_change":"2021-02-11T00:28:54.536787+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"e3cce55d-c182-4525-a567-52c86d5dc3db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6809","nonce":14492},{"type":"v1","addr":"172.21.15.95:6811","nonce":14492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6813","nonce":14492},{"type":"v1","addr":"172.21.15.95:6815","nonce":14492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6821","nonce":14492},{"type":"v1","addr":"172.21.15.95:6823","nonce":14492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6818","nonce":14492},{"type":"v1","addr":"172.21.15.95:6820","nonce":14492}]},"public_addr":"172.21.15.95:6811/14492","cluster_addr":"172.21.15.95:6815/14492","heartbeat_back_addr":"172.21.15.95:6823/14492","heartbeat_front_addr":"172.21.15.95:6820/14492","state":["exists","up"]},{"osd":1,"uuid":"618c16cc-ae3f-4158-8f89-2c2e7976780c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6800","nonce":14493},{"type":"v1","addr":"172.21.15.95:6801","nonce":14493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6802","nonce":14493},{"type":"v1","addr":"172.21.15.95:6803","nonce":14493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6806","nonce":14493},{"type":"v1","addr":"172.21.15.95:6807","nonce":14493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6804","nonce":14493},{"type":"v1","addr":"172.21.15.95:6805","nonce":14493}]},"public_addr":"172.21.15.95:6801/14493","cluster_addr":"172.21.15.95:6803/14493","heartbeat_back_addr":"172.21.15.95:6807/14493","heartbeat_front_addr":"172.21.15.95:6805/14493","state":["exists","up"]},{"osd":2,"uuid":"91671609-8e48-43aa-9b1f-f64b54a33d9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6808","nonce":14491},{"type":"v1","addr":"172.21.15.95:6810","nonce":14491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6812","nonce":14491},{"type":"v1","addr":"172.21.15.95:6814","nonce":14491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6819","nonce":14491},{"type":"v1","addr":"172.21.15.95:6822","nonce":14491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6816","nonce":14491},{"type":"v1","addr":"172.21.15.95:6817","nonce":14491}]},"public_addr":"172.21.15.95:6810/14491","cluster_addr":"172.21.15.95:6814/14491","heartbeat_back_addr":"172.21.15.95:6822/14491","heartbeat_front_addr":"172.21.15.95:6817/14491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:52.509433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T00:28:55.245 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-02-11T00:28:55.245 INFO:tasks.ceph:Creating RBD pool 2021-02-11T00:28:55.245 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph --cluster c2 osd pool create rbd 8 2021-02-11T00:28:56.557 INFO:teuthology.orchestra.run.smithi095.stderr:pool 'rbd' created 2021-02-11T00:28:56.566 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-02-11T00:28:57.556 INFO:teuthology.orchestra.run.smithi095.stderr:enabled application 'rbd' on pool 'rbd' 2021-02-11T00:28:57.571 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2021-02-11T00:28:57.571 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-02-11T00:28:57.572 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-02-11T00:28:57.871 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:57.872 INFO:teuthology.orchestra.run.smithi095.stdout:{"epoch":12,"fsid":"20d74388-aeac-4108-8e9a-c22b365ad363","created":"2021-02-11T00:28:46.956056+0000","modified":"2021-02-11T00:28:57.541114+0000","last_up_change":"2021-02-11T00:28:54.536787+0000","last_in_change":"2021-02-11T00:28:54.536787+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T00:28:54.548076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T00:28:55.544852+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e3cce55d-c182-4525-a567-52c86d5dc3db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6809","nonce":14492},{"type":"v1","addr":"172.21.15.95:6811","nonce":14492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6813","nonce":14492},{"type":"v1","addr":"172.21.15.95:6815","nonce":14492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6821","nonce":14492},{"type":"v1","addr":"172.21.15.95:6823","nonce":14492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6818","nonce":14492},{"type":"v1","addr":"172.21.15.95:6820","nonce":14492}]},"public_addr":"172.21.15.95:6811/14492","cluster_addr":"172.21.15.95:6815/14492","heartbeat_back_addr":"172.21.15.95:6823/14492","heartbeat_front_addr":"172.21.15.95:6820/14492","state":["exists","up"]},{"osd":1,"uuid":"618c16cc-ae3f-4158-8f89-2c2e7976780c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6800","nonce":14493},{"type":"v1","addr":"172.21.15.95:6801","nonce":14493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6802","nonce":14493},{"type":"v1","addr":"172.21.15.95:6803","nonce":14493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6806","nonce":14493},{"type":"v1","addr":"172.21.15.95:6807","nonce":14493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6804","nonce":14493},{"type":"v1","addr":"172.21.15.95:6805","nonce":14493}]},"public_addr":"172.21.15.95:6801/14493","cluster_addr":"172.21.15.95:6803/14493","heartbeat_back_addr":"172.21.15.95:6807/14493","heartbeat_front_addr":"172.21.15.95:6805/14493","state":["exists","up"]},{"osd":2,"uuid":"91671609-8e48-43aa-9b1f-f64b54a33d9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6808","nonce":14491},{"type":"v1","addr":"172.21.15.95:6810","nonce":14491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6812","nonce":14491},{"type":"v1","addr":"172.21.15.95:6814","nonce":14491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6819","nonce":14491},{"type":"v1","addr":"172.21.15.95:6822","nonce":14491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6816","nonce":14491},{"type":"v1","addr":"172.21.15.95:6817","nonce":14491}]},"public_addr":"172.21.15.95:6810/14491","cluster_addr":"172.21.15.95:6814/14491","heartbeat_back_addr":"172.21.15.95:6822/14491","heartbeat_front_addr":"172.21.15.95:6817/14491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:52.650890+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:52.509433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:52.755288+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T00:28:57.887 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-02-11T00:28:54.548076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-02-11T00:28:55.544852+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-11T00:28:57.887 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get device_health_metrics pg_num 2021-02-11T00:28:58.198 INFO:teuthology.orchestra.run.smithi095.stdout:pg_num: 1 2021-02-11T00:28:58.214 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get rbd pg_num 2021-02-11T00:28:58.518 INFO:teuthology.orchestra.run.smithi095.stdout:pg_num: 8 2021-02-11T00:28:58.532 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2021-02-11T00:28:58.532 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2021-02-11T00:28:58.533 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2021-02-11T00:28:58.862 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:58.876 INFO:teuthology.orchestra.run.smithi095.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6824","nonce":14305},{"type":"v1","addr":"172.21.15.95:6825","nonce":14305}]},"active_addr":"172.21.15.95:6825/14305","active_change":"2021-02-11T00:28:51.676861+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.95:0","nonce":3275572236}]},{"addrvec":[{"type":"v2","addr":"172.21.15.95:0","nonce":228394639}]}]}} 2021-02-11T00:28:58.878 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2021-02-11T00:28:58.878 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2021-02-11T00:28:58.879 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-02-11T00:28:59.175 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:59.175 INFO:teuthology.orchestra.run.smithi095.stdout:{"epoch":13,"fsid":"20d74388-aeac-4108-8e9a-c22b365ad363","created":"2021-02-11T00:28:46.956056+0000","modified":"2021-02-11T00:28:58.542698+0000","last_up_change":"2021-02-11T00:28:54.536787+0000","last_in_change":"2021-02-11T00:28:54.536787+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T00:28:54.548076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T00:28:55.544852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e3cce55d-c182-4525-a567-52c86d5dc3db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6809","nonce":14492},{"type":"v1","addr":"172.21.15.95:6811","nonce":14492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6813","nonce":14492},{"type":"v1","addr":"172.21.15.95:6815","nonce":14492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6821","nonce":14492},{"type":"v1","addr":"172.21.15.95:6823","nonce":14492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6818","nonce":14492},{"type":"v1","addr":"172.21.15.95:6820","nonce":14492}]},"public_addr":"172.21.15.95:6811/14492","cluster_addr":"172.21.15.95:6815/14492","heartbeat_back_addr":"172.21.15.95:6823/14492","heartbeat_front_addr":"172.21.15.95:6820/14492","state":["exists","up"]},{"osd":1,"uuid":"618c16cc-ae3f-4158-8f89-2c2e7976780c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6800","nonce":14493},{"type":"v1","addr":"172.21.15.95:6801","nonce":14493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6802","nonce":14493},{"type":"v1","addr":"172.21.15.95:6803","nonce":14493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6806","nonce":14493},{"type":"v1","addr":"172.21.15.95:6807","nonce":14493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6804","nonce":14493},{"type":"v1","addr":"172.21.15.95:6805","nonce":14493}]},"public_addr":"172.21.15.95:6801/14493","cluster_addr":"172.21.15.95:6803/14493","heartbeat_back_addr":"172.21.15.95:6807/14493","heartbeat_front_addr":"172.21.15.95:6805/14493","state":["exists","up"]},{"osd":2,"uuid":"91671609-8e48-43aa-9b1f-f64b54a33d9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6808","nonce":14491},{"type":"v1","addr":"172.21.15.95:6810","nonce":14491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6812","nonce":14491},{"type":"v1","addr":"172.21.15.95:6814","nonce":14491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6819","nonce":14491},{"type":"v1","addr":"172.21.15.95:6822","nonce":14491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6816","nonce":14491},{"type":"v1","addr":"172.21.15.95:6817","nonce":14491}]},"public_addr":"172.21.15.95:6810/14491","cluster_addr":"172.21.15.95:6814/14491","heartbeat_back_addr":"172.21.15.95:6822/14491","heartbeat_front_addr":"172.21.15.95:6817/14491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:52.650890+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:52.509433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:52.755288+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T00:28:59.189 INFO:tasks.ceph.ceph_manager.c2:all up! 2021-02-11T00:28:59.190 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-02-11T00:28:59.560 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:28:59.561 INFO:teuthology.orchestra.run.smithi095.stdout:{"epoch":13,"fsid":"20d74388-aeac-4108-8e9a-c22b365ad363","created":"2021-02-11T00:28:46.956056+0000","modified":"2021-02-11T00:28:58.542698+0000","last_up_change":"2021-02-11T00:28:54.536787+0000","last_in_change":"2021-02-11T00:28:54.536787+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T00:28:54.548076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T00:28:55.544852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e3cce55d-c182-4525-a567-52c86d5dc3db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6809","nonce":14492},{"type":"v1","addr":"172.21.15.95:6811","nonce":14492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6813","nonce":14492},{"type":"v1","addr":"172.21.15.95:6815","nonce":14492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6821","nonce":14492},{"type":"v1","addr":"172.21.15.95:6823","nonce":14492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6818","nonce":14492},{"type":"v1","addr":"172.21.15.95:6820","nonce":14492}]},"public_addr":"172.21.15.95:6811/14492","cluster_addr":"172.21.15.95:6815/14492","heartbeat_back_addr":"172.21.15.95:6823/14492","heartbeat_front_addr":"172.21.15.95:6820/14492","state":["exists","up"]},{"osd":1,"uuid":"618c16cc-ae3f-4158-8f89-2c2e7976780c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6800","nonce":14493},{"type":"v1","addr":"172.21.15.95:6801","nonce":14493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6802","nonce":14493},{"type":"v1","addr":"172.21.15.95:6803","nonce":14493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6806","nonce":14493},{"type":"v1","addr":"172.21.15.95:6807","nonce":14493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6804","nonce":14493},{"type":"v1","addr":"172.21.15.95:6805","nonce":14493}]},"public_addr":"172.21.15.95:6801/14493","cluster_addr":"172.21.15.95:6803/14493","heartbeat_back_addr":"172.21.15.95:6807/14493","heartbeat_front_addr":"172.21.15.95:6805/14493","state":["exists","up"]},{"osd":2,"uuid":"91671609-8e48-43aa-9b1f-f64b54a33d9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6808","nonce":14491},{"type":"v1","addr":"172.21.15.95:6810","nonce":14491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6812","nonce":14491},{"type":"v1","addr":"172.21.15.95:6814","nonce":14491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6819","nonce":14491},{"type":"v1","addr":"172.21.15.95:6822","nonce":14491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6816","nonce":14491},{"type":"v1","addr":"172.21.15.95:6817","nonce":14491}]},"public_addr":"172.21.15.95:6810/14491","cluster_addr":"172.21.15.95:6814/14491","heartbeat_back_addr":"172.21.15.95:6822/14491","heartbeat_front_addr":"172.21.15.95:6817/14491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:52.650890+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:52.509433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:52.755288+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T00:28:59.580 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2021-02-11T00:28:59.740 INFO:teuthology.orchestra.run.smithi095.stdout:38654705667 2021-02-11T00:28:59.740 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2021-02-11T00:28:59.901 INFO:teuthology.orchestra.run.smithi095.stdout:34359738372 2021-02-11T00:28:59.901 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2021-02-11T00:29:00.087 INFO:teuthology.orchestra.run.smithi095.stdout:38654705667 2021-02-11T00:29:00.088 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-02-11T00:29:00.396 INFO:teuthology.orchestra.run.smithi095.stdout:38654705666 2021-02-11T00:29:00.412 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.0 2021-02-11T00:29:01.412 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-02-11T00:29:01.718 INFO:teuthology.orchestra.run.smithi095.stdout:38654705666 2021-02-11T00:29:01.733 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.0 2021-02-11T00:29:02.733 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-02-11T00:29:03.049 INFO:teuthology.orchestra.run.smithi095.stdout:38654705668 2021-02-11T00:29:03.064 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705668 for osd.0 2021-02-11T00:29:03.064 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2021-02-11T00:29:03.380 INFO:teuthology.orchestra.run.smithi095.stdout:34359738372 2021-02-11T00:29:03.394 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.1 2021-02-11T00:29:03.394 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2021-02-11T00:29:03.681 INFO:teuthology.orchestra.run.smithi095.stdout:38654705668 2021-02-11T00:29:03.694 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705668 for osd.2 2021-02-11T00:29:03.694 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2021-02-11T00:29:03.694 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-02-11T00:29:03.987 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:29:03.989 INFO:teuthology.orchestra.run.smithi095.stderr:dumped all 2021-02-11T00:29:04.001 INFO:teuthology.orchestra.run.smithi095.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-02-11T00:29:03.683505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14680,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":14336,"kb_avail":314558120,"statfs":{"total":322122547200,"available":322107514880,"internally_reserved":0,"allocated":327680,"data_stored":88587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14680064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.564507+0000","last_change":"2021-02-11T00:28:57.562765+0000","last_active":"2021-02-11T00:28:57.564507+0000","last_peered":"2021-02-11T00:28:57.564507+0000","last_clean":"2021-02-11T00:28:57.564507+0000","last_became_active":"2021-02-11T00:28:57.562399+0000","last_became_peered":"2021-02-11T00:28:57.562399+0000","last_unstale":"2021-02-11T00:28:57.564507+0000","last_undegraded":"2021-02-11T00:28:57.564507+0000","last_fullsized":"2021-02-11T00:28:57.564507+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_clean_scrub_stamp":"2021-02-11T00:28:56.539933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.564479+0000","last_change":"2021-02-11T00:28:57.561294+0000","last_active":"2021-02-11T00:28:57.564479+0000","last_peered":"2021-02-11T00:28:57.564479+0000","last_clean":"2021-02-11T00:28:57.564479+0000","last_became_active":"2021-02-11T00:28:57.560960+0000","last_became_peered":"2021-02-11T00:28:57.560960+0000","last_unstale":"2021-02-11T00:28:57.564479+0000","last_undegraded":"2021-02-11T00:28:57.564479+0000","last_fullsized":"2021-02-11T00:28:57.564479+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_clean_scrub_stamp":"2021-02-11T00:28:56.539933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.558405+0000","last_change":"2021-02-11T00:28:57.557693+0000","last_active":"2021-02-11T00:28:57.558405+0000","last_peered":"2021-02-11T00:28:57.558405+0000","last_clean":"2021-02-11T00:28:57.558405+0000","last_became_active":"2021-02-11T00:28:57.557343+0000","last_became_peered":"2021-02-11T00:28:57.557343+0000","last_unstale":"2021-02-11T00:28:57.558405+0000","last_undegraded":"2021-02-11T00:28:57.558405+0000","last_fullsized":"2021-02-11T00:28:57.558405+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_clean_scrub_stamp":"2021-02-11T00:28:56.539933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.559093+0000","last_change":"2021-02-11T00:28:57.559093+0000","last_active":"2021-02-11T00:28:57.559093+0000","last_peered":"2021-02-11T00:28:57.559093+0000","last_clean":"2021-02-11T00:28:57.559093+0000","last_became_active":"2021-02-11T00:28:57.558806+0000","last_became_peered":"2021-02-11T00:28:57.558806+0000","last_unstale":"2021-02-11T00:28:57.559093+0000","last_undegraded":"2021-02-11T00:28:57.559093+0000","last_fullsized":"2021-02-11T00:28:57.559093+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_clean_scrub_stamp":"2021-02-11T00:28:56.539933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.553498+0000","last_change":"2021-02-11T00:28:56.557584+0000","last_active":"2021-02-11T00:28:57.553498+0000","last_peered":"2021-02-11T00:28:57.553498+0000","last_clean":"2021-02-11T00:28:57.553498+0000","last_became_active":"2021-02-11T00:28:56.557255+0000","last_became_peered":"2021-02-11T00:28:56.557255+0000","last_unstale":"2021-02-11T00:28:57.553498+0000","last_undegraded":"2021-02-11T00:28:57.553498+0000","last_fullsized":"2021-02-11T00:28:57.553498+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:55.538617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:55.538617+0000","last_clean_scrub_stamp":"2021-02-11T00:28:55.538617+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.563451+0000","last_change":"2021-02-11T00:28:57.560266+0000","last_active":"2021-02-11T00:28:57.563451+0000","last_peered":"2021-02-11T00:28:57.563451+0000","last_clean":"2021-02-11T00:28:57.563451+0000","last_became_active":"2021-02-11T00:28:57.559942+0000","last_became_peered":"2021-02-11T00:28:57.559942+0000","last_unstale":"2021-02-11T00:28:57.563451+0000","last_undegraded":"2021-02-11T00:28:57.563451+0000","last_fullsized":"2021-02-11T00:28:57.563451+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_clean_scrub_stamp":"2021-02-11T00:28:56.539933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.556974+0000","last_change":"2021-02-11T00:28:57.556974+0000","last_active":"2021-02-11T00:28:57.556974+0000","last_peered":"2021-02-11T00:28:57.556974+0000","last_clean":"2021-02-11T00:28:57.556974+0000","last_became_active":"2021-02-11T00:28:57.556313+0000","last_became_peered":"2021-02-11T00:28:57.556313+0000","last_unstale":"2021-02-11T00:28:57.556974+0000","last_undegraded":"2021-02-11T00:28:57.556974+0000","last_fullsized":"2021-02-11T00:28:57.556974+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_clean_scrub_stamp":"2021-02-11T00:28:56.539933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.559279+0000","last_change":"2021-02-11T00:28:57.559279+0000","last_active":"2021-02-11T00:28:57.559279+0000","last_peered":"2021-02-11T00:28:57.559279+0000","last_clean":"2021-02-11T00:28:57.559279+0000","last_became_active":"2021-02-11T00:28:57.558675+0000","last_became_peered":"2021-02-11T00:28:57.558675+0000","last_unstale":"2021-02-11T00:28:57.559279+0000","last_undegraded":"2021-02-11T00:28:57.559279+0000","last_fullsized":"2021-02-11T00:28:57.559279+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_clean_scrub_stamp":"2021-02-11T00:28:56.539933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.559216+0000","last_change":"2021-02-11T00:28:57.557893+0000","last_active":"2021-02-11T00:28:57.559216+0000","last_peered":"2021-02-11T00:28:57.559216+0000","last_clean":"2021-02-11T00:28:57.559216+0000","last_became_active":"2021-02-11T00:28:57.557310+0000","last_became_peered":"2021-02-11T00:28:57.557310+0000","last_unstale":"2021-02-11T00:28:57.559216+0000","last_undegraded":"2021-02-11T00:28:57.559216+0000","last_fullsized":"2021-02-11T00:28:57.559216+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_clean_scrub_stamp":"2021-02-11T00:28:56.539933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4848,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":104852752,"statfs":{"total":107374182400,"available":107369218048,"internally_reserved":0,"allocated":106496,"data_stored":28358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":31871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":28358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]}]}]}} 2021-02-11T00:29:04.002 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-02-11T00:29:04.295 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:29:04.296 INFO:teuthology.orchestra.run.smithi095.stderr:dumped all 2021-02-11T00:29:04.308 INFO:teuthology.orchestra.run.smithi095.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-02-11T00:29:03.683505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14680,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":14336,"kb_avail":314558120,"statfs":{"total":322122547200,"available":322107514880,"internally_reserved":0,"allocated":327680,"data_stored":88587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14680064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.564507+0000","last_change":"2021-02-11T00:28:57.562765+0000","last_active":"2021-02-11T00:28:57.564507+0000","last_peered":"2021-02-11T00:28:57.564507+0000","last_clean":"2021-02-11T00:28:57.564507+0000","last_became_active":"2021-02-11T00:28:57.562399+0000","last_became_peered":"2021-02-11T00:28:57.562399+0000","last_unstale":"2021-02-11T00:28:57.564507+0000","last_undegraded":"2021-02-11T00:28:57.564507+0000","last_fullsized":"2021-02-11T00:28:57.564507+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_clean_scrub_stamp":"2021-02-11T00:28:56.539933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.564479+0000","last_change":"2021-02-11T00:28:57.561294+0000","last_active":"2021-02-11T00:28:57.564479+0000","last_peered":"2021-02-11T00:28:57.564479+0000","last_clean":"2021-02-11T00:28:57.564479+0000","last_became_active":"2021-02-11T00:28:57.560960+0000","last_became_peered":"2021-02-11T00:28:57.560960+0000","last_unstale":"2021-02-11T00:28:57.564479+0000","last_undegraded":"2021-02-11T00:28:57.564479+0000","last_fullsized":"2021-02-11T00:28:57.564479+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_clean_scrub_stamp":"2021-02-11T00:28:56.539933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.558405+0000","last_change":"2021-02-11T00:28:57.557693+0000","last_active":"2021-02-11T00:28:57.558405+0000","last_peered":"2021-02-11T00:28:57.558405+0000","last_clean":"2021-02-11T00:28:57.558405+0000","last_became_active":"2021-02-11T00:28:57.557343+0000","last_became_peered":"2021-02-11T00:28:57.557343+0000","last_unstale":"2021-02-11T00:28:57.558405+0000","last_undegraded":"2021-02-11T00:28:57.558405+0000","last_fullsized":"2021-02-11T00:28:57.558405+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_clean_scrub_stamp":"2021-02-11T00:28:56.539933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.559093+0000","last_change":"2021-02-11T00:28:57.559093+0000","last_active":"2021-02-11T00:28:57.559093+0000","last_peered":"2021-02-11T00:28:57.559093+0000","last_clean":"2021-02-11T00:28:57.559093+0000","last_became_active":"2021-02-11T00:28:57.558806+0000","last_became_peered":"2021-02-11T00:28:57.558806+0000","last_unstale":"2021-02-11T00:28:57.559093+0000","last_undegraded":"2021-02-11T00:28:57.559093+0000","last_fullsized":"2021-02-11T00:28:57.559093+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_clean_scrub_stamp":"2021-02-11T00:28:56.539933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.553498+0000","last_change":"2021-02-11T00:28:56.557584+0000","last_active":"2021-02-11T00:28:57.553498+0000","last_peered":"2021-02-11T00:28:57.553498+0000","last_clean":"2021-02-11T00:28:57.553498+0000","last_became_active":"2021-02-11T00:28:56.557255+0000","last_became_peered":"2021-02-11T00:28:56.557255+0000","last_unstale":"2021-02-11T00:28:57.553498+0000","last_undegraded":"2021-02-11T00:28:57.553498+0000","last_fullsized":"2021-02-11T00:28:57.553498+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:55.538617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:55.538617+0000","last_clean_scrub_stamp":"2021-02-11T00:28:55.538617+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.563451+0000","last_change":"2021-02-11T00:28:57.560266+0000","last_active":"2021-02-11T00:28:57.563451+0000","last_peered":"2021-02-11T00:28:57.563451+0000","last_clean":"2021-02-11T00:28:57.563451+0000","last_became_active":"2021-02-11T00:28:57.559942+0000","last_became_peered":"2021-02-11T00:28:57.559942+0000","last_unstale":"2021-02-11T00:28:57.563451+0000","last_undegraded":"2021-02-11T00:28:57.563451+0000","last_fullsized":"2021-02-11T00:28:57.563451+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_clean_scrub_stamp":"2021-02-11T00:28:56.539933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.556974+0000","last_change":"2021-02-11T00:28:57.556974+0000","last_active":"2021-02-11T00:28:57.556974+0000","last_peered":"2021-02-11T00:28:57.556974+0000","last_clean":"2021-02-11T00:28:57.556974+0000","last_became_active":"2021-02-11T00:28:57.556313+0000","last_became_peered":"2021-02-11T00:28:57.556313+0000","last_unstale":"2021-02-11T00:28:57.556974+0000","last_undegraded":"2021-02-11T00:28:57.556974+0000","last_fullsized":"2021-02-11T00:28:57.556974+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_clean_scrub_stamp":"2021-02-11T00:28:56.539933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.559279+0000","last_change":"2021-02-11T00:28:57.559279+0000","last_active":"2021-02-11T00:28:57.559279+0000","last_peered":"2021-02-11T00:28:57.559279+0000","last_clean":"2021-02-11T00:28:57.559279+0000","last_became_active":"2021-02-11T00:28:57.558675+0000","last_became_peered":"2021-02-11T00:28:57.558675+0000","last_unstale":"2021-02-11T00:28:57.559279+0000","last_undegraded":"2021-02-11T00:28:57.559279+0000","last_fullsized":"2021-02-11T00:28:57.559279+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_clean_scrub_stamp":"2021-02-11T00:28:56.539933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-11T00:28:57.559216+0000","last_change":"2021-02-11T00:28:57.557893+0000","last_active":"2021-02-11T00:28:57.559216+0000","last_peered":"2021-02-11T00:28:57.559216+0000","last_clean":"2021-02-11T00:28:57.559216+0000","last_became_active":"2021-02-11T00:28:57.557310+0000","last_became_peered":"2021-02-11T00:28:57.557310+0000","last_unstale":"2021-02-11T00:28:57.559216+0000","last_undegraded":"2021-02-11T00:28:57.559216+0000","last_fullsized":"2021-02-11T00:28:57.559216+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-11T00:28:56.539933+0000","last_clean_scrub_stamp":"2021-02-11T00:28:56.539933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4848,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":104852752,"statfs":{"total":107374182400,"available":107369218048,"internally_reserved":0,"allocated":106496,"data_stored":28358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":31871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":28358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]}]}]}} 2021-02-11T00:29:04.309 INFO:tasks.ceph.ceph_manager.c2:clean! 2021-02-11T00:29:04.309 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2021-02-11T00:29:04.309 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2021-02-11T00:29:04.309 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2021-02-11T00:29:04.646 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T00:29:04.659 INFO:teuthology.orchestra.run.smithi095.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-02-11T00:29:04.659 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2021-02-11T00:29:04.659 INFO:teuthology.run_tasks:Running task rgw... 2021-02-11T00:29:04.685 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2021-02-11T00:29:04.685 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2021-02-11T00:29:04.686 INFO:tasks.rgw:Creating data pools 2021-02-11T00:29:04.686 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2021-02-11T00:29:04.686 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-02-11T00:29:05.851 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'default.rgw.buckets.data' created 2021-02-11T00:29:05.867 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-02-11T00:29:06.862 INFO:teuthology.orchestra.run.smithi057.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-11T00:29:06.883 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-02-11T00:29:07.864 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'default.rgw.buckets.index' created 2021-02-11T00:29:07.886 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-02-11T00:29:08.884 INFO:teuthology.orchestra.run.smithi057.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-11T00:29:08.900 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2021-02-11T00:29:08.900 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-02-11T00:29:09.202 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'default.rgw.buckets.data' already exists 2021-02-11T00:29:09.214 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-02-11T00:29:09.877 INFO:teuthology.orchestra.run.smithi057.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-11T00:29:09.892 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-02-11T00:29:10.204 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'default.rgw.buckets.index' already exists 2021-02-11T00:29:10.215 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-02-11T00:29:10.888 INFO:teuthology.orchestra.run.smithi057.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-11T00:29:10.904 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2021-02-11T00:29:10.904 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-02-11T00:29:11.983 INFO:teuthology.orchestra.run.smithi095.stderr:pool 'default.rgw.buckets.data' created 2021-02-11T00:29:11.994 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-02-11T00:29:12.998 INFO:teuthology.orchestra.run.smithi095.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-11T00:29:13.016 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-02-11T00:29:13.987 INFO:teuthology.orchestra.run.smithi095.stderr:pool 'default.rgw.buckets.index' created 2021-02-11T00:29:14.012 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-02-11T00:29:15.019 INFO:teuthology.orchestra.run.smithi095.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-11T00:29:15.036 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2021-02-11T00:29:15.036 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-02-11T00:29:15.330 INFO:teuthology.orchestra.run.smithi095.stderr:pool 'default.rgw.buckets.data' already exists 2021-02-11T00:29:15.341 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-02-11T00:29:16.007 INFO:teuthology.orchestra.run.smithi095.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-02-11T00:29:16.021 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-02-11T00:29:16.323 INFO:teuthology.orchestra.run.smithi095.stderr:pool 'default.rgw.buckets.index' already exists 2021-02-11T00:29:16.335 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-02-11T00:29:17.019 INFO:teuthology.orchestra.run.smithi095.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-02-11T00:29:17.032 DEBUG:tasks.rgw:Pools created 2021-02-11T00:29:17.032 INFO:tasks.rgw:Configuring compression type = random 2021-02-11T00:29:17.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2021-02-11T00:29:17.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2021-02-11T00:29:17.032 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2021-02-11T00:29:17.070 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:29:17.070 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:29:17.081 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:17.077+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:18.827 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.821+0000 7fe7089ef080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:18.828 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.821+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:18.829 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.825+0000 7fe7089ef080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:18.829 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.825+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:18.830 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.825+0000 7fe7089ef080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:18.830 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.825+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:18.830 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.825+0000 7fe7089ef080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:18.853 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.849+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:18.854 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.849+0000 7fe7089ef080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:18.854 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.849+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:18.854 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.849+0000 7fe7089ef080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:18.854 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.849+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:18.855 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.849+0000 7fe7089ef080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:18.855 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.849+0000 7fe7089ef080 10 cannot find current period zonegroup using local zonegroup 2021-02-11T00:29:18.855 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.849+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:18.855 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.849+0000 7fe7089ef080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:18.855 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.849+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:18.855 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.849+0000 7fe7089ef080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:18.856 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.849+0000 7fe7089ef080 10 Creating default zonegroup 2021-02-11T00:29:18.856 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.849+0000 7fe7089ef080 20 get_system_obj_state: rctx=0x7fff1cbe43b0 obj=:.pools.avail state=0x56434a0ca1f0 s->prefetch_data=0 2021-02-11T00:29:18.856 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.849+0000 7fe7089ef080 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-02-11T00:29:18.873 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.869+0000 7fe7089ef080 10 WARNING: store->list_zones() returned r=-2 2021-02-11T00:29:18.873 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.869+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:18.874 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.869+0000 7fe7089ef080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:18.880 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.877+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:18.880 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.877+0000 7fe7089ef080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:18.881 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.877+0000 7fe7089ef080 10 could not read realm id: (2) No such file or directory 2021-02-11T00:29:18.881 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.877+0000 7fe7089ef080 10 WARNING: failed to set zone as default, r=-22 2021-02-11T00:29:18.881 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.877+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:18.881 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.877+0000 7fe7089ef080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:18.886 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.881+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:18.888 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.885+0000 7fe7089ef080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T00:29:18.888 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.885+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:18.891 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.885+0000 7fe7089ef080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T00:29:18.891 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.885+0000 7fe7089ef080 20 zonegroup default 2021-02-11T00:29:18.891 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.885+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:18.891 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.885+0000 7fe7089ef080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:18.891 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.885+0000 7fe7089ef080 10 Cannot find current period zone using local zone 2021-02-11T00:29:18.891 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.885+0000 7fe7089ef080 10 Using default name default 2021-02-11T00:29:18.891 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.885+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:18.892 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.889+0000 7fe7089ef080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:18.892 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.889+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:18.893 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.889+0000 7fe7089ef080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T00:29:18.893 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.889+0000 7fe7089ef080 20 zone default found 2021-02-11T00:29:18.893 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:18.889+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:20.864 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:20.857+0000 7fe7089ef080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:20.864 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:20.857+0000 7fe7089ef080 20 rados->read ofs=0 len=0 2021-02-11T00:29:20.865 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:20.861+0000 7fe7089ef080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:20.865 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:20.861+0000 7fe7089ef080 20 started sync module instance, tier type = 2021-02-11T00:29:20.865 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:20.861+0000 7fe7089ef080 20 started zone id=23a8e993-ccac-4be4-a406-474148bcf8be (name=default) with tier type = 2021-02-11T00:29:22.937 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:22.933+0000 7fe7089ef080 20 add_watcher() i=0 2021-02-11T00:29:22.937 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:22.933+0000 7fe7089ef080 20 add_watcher() i=1 2021-02-11T00:29:22.938 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:22.933+0000 7fe7089ef080 20 add_watcher() i=2 2021-02-11T00:29:22.938 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:22.933+0000 7fe7089ef080 20 add_watcher() i=3 2021-02-11T00:29:22.938 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:22.933+0000 7fe7089ef080 20 add_watcher() i=4 2021-02-11T00:29:22.938 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:22.933+0000 7fe7089ef080 20 add_watcher() i=5 2021-02-11T00:29:22.938 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:22.933+0000 7fe7089ef080 20 add_watcher() i=6 2021-02-11T00:29:22.939 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:22.933+0000 7fe7089ef080 20 add_watcher() i=7 2021-02-11T00:29:22.939 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:22.933+0000 7fe7089ef080 2 all 8 watchers are set, enabling cache 2021-02-11T00:29:22.940 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:22.937+0000 7fe7089ef080 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(optional_yield):423: oid=data_loggenerations_metadata not found 2021-02-11T00:29:22.954 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:22.949+0000 7fe6c57ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T00:29:22.954 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:22.949+0000 7fe7089ef080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T00:29:22.954 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:22.949+0000 7fe7089ef080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T00:29:26.891 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:26.885+0000 7fe7089ef080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-02-11T00:29:27.014 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.009+0000 7fe627fff700 20 reqs_thread_entry: start 2021-02-11T00:29:27.014 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.009+0000 7fe7089ef080 20 init_complete bucket index max shards: 11 2021-02-11T00:29:27.014 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.009+0000 7fe625ffb700 20 reqs_thread_entry: start 2021-02-11T00:29:27.014 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.009+0000 7fe7089ef080 10 rgw notify: Started notification manager with: 1 workers 2021-02-11T00:29:27.014 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.009+0000 7fe613fff700 20 rgw notify: INFO: next queues processing will happen at: Thu Feb 11 00:29:57 2021 2021-02-11T00:29:27.014 INFO:teuthology.orchestra.run.smithi057.stderr: 2021-02-11T00:29:27.014 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.009+0000 7fe6137fe700 20 reqs_thread_entry: start 2021-02-11T00:29:27.045 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.041+0000 7fe7089ef080 20 remove_watcher() i=0 2021-02-11T00:29:27.045 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.041+0000 7fe7089ef080 2 removed watcher, disabling cache 2021-02-11T00:29:27.048 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.045+0000 7fe7089ef080 20 remove_watcher() i=1 2021-02-11T00:29:27.051 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.045+0000 7fe7089ef080 20 remove_watcher() i=2 2021-02-11T00:29:27.054 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.049+0000 7fe7089ef080 20 remove_watcher() i=3 2021-02-11T00:29:27.057 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.053+0000 7fe7089ef080 20 remove_watcher() i=4 2021-02-11T00:29:27.059 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.053+0000 7fe7089ef080 20 remove_watcher() i=5 2021-02-11T00:29:27.062 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.057+0000 7fe7089ef080 20 remove_watcher() i=6 2021-02-11T00:29:27.065 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.061+0000 7fe7089ef080 20 remove_watcher() i=7 2021-02-11T00:29:27.067 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.061+0000 7fe6c57ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T00:29:27.076 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2021-02-11T00:29:27.077 DEBUG:tasks.util.rgw: json result: [] 2021-02-11T00:29:27.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T00:29:27.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T00:29:27.077 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-11T00:29:27.159 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:29:27.160 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:29:27.173 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.169+0000 7fb63f0e9080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T00:29:27.173 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.169+0000 7fb63f0e9080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T00:29:27.179 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.173+0000 7fb573fff700 20 reqs_thread_entry: start 2021-02-11T00:29:27.179 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.173+0000 7fb63f0e9080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.180 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.177+0000 7fb63f0e9080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:27.181 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.177+0000 7fb63f0e9080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.181 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.177+0000 7fb63f0e9080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T00:29:27.182 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.177+0000 7fb63f0e9080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.182 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.177+0000 7fb63f0e9080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:27.182 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.177+0000 7fb63f0e9080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.183 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.177+0000 7fb63f0e9080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:27.183 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.177+0000 7fb63f0e9080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.183 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.181+0000 7fb63f0e9080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T00:29:27.204 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"23a8e993-ccac-4be4-a406-474148bcf8be","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2021-02-11T00:29:27.204 DEBUG:tasks.util.rgw: json result: {'id': '23a8e993-ccac-4be4-a406-474148bcf8be', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-02-11T00:29:27.204 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2021-02-11T00:29:27.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2021-02-11T00:29:27.205 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2021-02-11T00:29:27.283 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:29:27.283 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:29:27.292 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.289+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.295 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.289+0000 7fdfeb8bd080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:27.295 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.289+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.296 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.293+0000 7fdfeb8bd080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:27.296 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.293+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.297 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.293+0000 7fdfeb8bd080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:27.297 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.293+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.297 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.293+0000 7fdfeb8bd080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:27.324 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.321+0000 7fdfeb8bd080 20 RGWRados::pool_iterate: got zone_info.23a8e993-ccac-4be4-a406-474148bcf8be 2021-02-11T00:29:27.324 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.321+0000 7fdfeb8bd080 20 RGWRados::pool_iterate: got zonegroup_info.b73141ab-56f9-4dee-9738-7eea05b1156d 2021-02-11T00:29:27.324 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.321+0000 7fdfeb8bd080 20 RGWRados::pool_iterate: got zone_names.default 2021-02-11T00:29:27.324 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.321+0000 7fdfeb8bd080 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-02-11T00:29:27.324 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.321+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.324 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.321+0000 7fdfeb8bd080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:27.325 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.321+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.325 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.321+0000 7fdfeb8bd080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T00:29:27.325 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.321+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.326 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.321+0000 7fdfeb8bd080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:27.326 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.321+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.327 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.321+0000 7fdfeb8bd080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T00:29:27.327 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.321+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.327 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.325+0000 7fdfeb8bd080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:27.328 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.325+0000 7fdfeb8bd080 10 cannot find current period zonegroup using local zonegroup 2021-02-11T00:29:27.328 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.325+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.328 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.325+0000 7fdfeb8bd080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:27.328 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.325+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.329 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.325+0000 7fdfeb8bd080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:27.329 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.325+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.330 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.325+0000 7fdfeb8bd080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T00:29:27.330 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.325+0000 7fdfeb8bd080 20 zonegroup default 2021-02-11T00:29:27.330 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.325+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.331 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.325+0000 7fdfeb8bd080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:27.331 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.325+0000 7fdfeb8bd080 10 Cannot find current period zone using local zone 2021-02-11T00:29:27.331 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.325+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.331 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.329+0000 7fdfeb8bd080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:27.331 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.329+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.332 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.329+0000 7fdfeb8bd080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:27.332 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.329+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.333 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.329+0000 7fdfeb8bd080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T00:29:27.333 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.329+0000 7fdfeb8bd080 20 zone default found 2021-02-11T00:29:27.333 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.329+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.334 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.329+0000 7fdfeb8bd080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:27.334 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.329+0000 7fdfeb8bd080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.334 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.329+0000 7fdfeb8bd080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:27.334 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.329+0000 7fdfeb8bd080 20 started sync module instance, tier type = 2021-02-11T00:29:27.335 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.329+0000 7fdfeb8bd080 20 started zone id=23a8e993-ccac-4be4-a406-474148bcf8be (name=default) with tier type = 2021-02-11T00:29:27.384 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.381+0000 7fdfeb8bd080 20 add_watcher() i=0 2021-02-11T00:29:27.384 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.381+0000 7fdfeb8bd080 20 add_watcher() i=1 2021-02-11T00:29:27.384 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.381+0000 7fdfeb8bd080 20 add_watcher() i=2 2021-02-11T00:29:27.384 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.381+0000 7fdfeb8bd080 20 add_watcher() i=3 2021-02-11T00:29:27.384 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.381+0000 7fdfeb8bd080 20 add_watcher() i=4 2021-02-11T00:29:27.384 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.381+0000 7fdfeb8bd080 20 add_watcher() i=5 2021-02-11T00:29:27.384 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.381+0000 7fdfeb8bd080 20 add_watcher() i=6 2021-02-11T00:29:27.387 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.381+0000 7fdfeb8bd080 20 add_watcher() i=7 2021-02-11T00:29:27.387 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.381+0000 7fdfeb8bd080 2 all 8 watchers are set, enabling cache 2021-02-11T00:29:27.391 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.385+0000 7fdf23fff700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T00:29:27.391 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.385+0000 7fdfeb8bd080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T00:29:27.391 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.385+0000 7fdfeb8bd080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T00:29:27.496 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.493+0000 7fdf12ffd700 20 reqs_thread_entry: start 2021-02-11T00:29:27.496 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.493+0000 7fdf10ff9700 20 reqs_thread_entry: start 2021-02-11T00:29:27.496 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.493+0000 7fdfeb8bd080 20 init_complete bucket index max shards: 11 2021-02-11T00:29:27.497 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.493+0000 7fdfeb8bd080 10 rgw notify: Started notification manager with: 1 workers 2021-02-11T00:29:27.497 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.493+0000 7fdf0a7fc700 20 reqs_thread_entry: start 2021-02-11T00:29:27.497 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.493+0000 7fdf0affd700 20 rgw notify: INFO: next queues processing will happen at: Thu Feb 11 00:29:57 2021 2021-02-11T00:29:27.497 INFO:teuthology.orchestra.run.smithi057.stderr: 2021-02-11T00:29:27.528 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.525+0000 7fdfeb8bd080 20 remove_watcher() i=0 2021-02-11T00:29:27.528 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.525+0000 7fdfeb8bd080 2 removed watcher, disabling cache 2021-02-11T00:29:27.531 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.525+0000 7fdfeb8bd080 20 remove_watcher() i=1 2021-02-11T00:29:27.534 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.529+0000 7fdfeb8bd080 20 remove_watcher() i=2 2021-02-11T00:29:27.537 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.533+0000 7fdfeb8bd080 20 remove_watcher() i=3 2021-02-11T00:29:27.540 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.537+0000 7fdfeb8bd080 20 remove_watcher() i=4 2021-02-11T00:29:27.542 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.537+0000 7fdfeb8bd080 20 remove_watcher() i=5 2021-02-11T00:29:27.545 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.541+0000 7fdfeb8bd080 20 remove_watcher() i=6 2021-02-11T00:29:27.548 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.545+0000 7fdfeb8bd080 20 remove_watcher() i=7 2021-02-11T00:29:27.550 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.545+0000 7fdf23fff700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T00:29:27.560 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2021-02-11T00:29:27.560 DEBUG:tasks.util.rgw: json result: [] 2021-02-11T00:29:27.560 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T00:29:27.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T00:29:27.560 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-11T00:29:27.647 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:29:27.647 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:29:27.658 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.653+0000 7f418967c080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T00:29:27.658 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.653+0000 7f418967c080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T00:29:27.666 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.661+0000 7f40be7fc700 20 reqs_thread_entry: start 2021-02-11T00:29:27.667 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.661+0000 7f418967c080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.668 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.665+0000 7f418967c080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:27.668 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.665+0000 7f418967c080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.669 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.665+0000 7f418967c080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T00:29:27.669 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.665+0000 7f418967c080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.670 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.665+0000 7f418967c080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:27.670 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.665+0000 7f418967c080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.670 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.665+0000 7f418967c080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:27.670 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.665+0000 7f418967c080 20 rados->read ofs=0 len=0 2021-02-11T00:29:27.671 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:27.665+0000 7f418967c080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T00:29:27.683 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"23a8e993-ccac-4be4-a406-474148bcf8be","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2021-02-11T00:29:27.684 DEBUG:tasks.util.rgw: json result: {'id': '23a8e993-ccac-4be4-a406-474148bcf8be', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-02-11T00:29:27.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2021-02-11T00:29:27.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2021-02-11T00:29:27.684 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2021-02-11T00:29:27.719 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:29:27.719 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:29:27.732 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:27.728+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:28.978 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:28.972+0000 7f3fff610080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:28.978 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:28.972+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:28.979 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:28.976+0000 7f3fff610080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:28.979 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:28.976+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:28.981 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:28.976+0000 7f3fff610080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:28.981 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:28.976+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:28.981 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:28.976+0000 7f3fff610080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:29.004 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.000+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:29.005 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.000+0000 7f3fff610080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:29.005 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.000+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:29.005 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.000+0000 7f3fff610080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:29.005 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.000+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:29.005 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.000+0000 7f3fff610080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:29.006 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.000+0000 7f3fff610080 10 cannot find current period zonegroup using local zonegroup 2021-02-11T00:29:29.006 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.000+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:29.006 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.000+0000 7f3fff610080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:29.006 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.000+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:29.007 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.004+0000 7f3fff610080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:29.007 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.004+0000 7f3fff610080 10 Creating default zonegroup 2021-02-11T00:29:29.007 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.004+0000 7f3fff610080 20 get_system_obj_state: rctx=0x7ffc4d1c0250 obj=:.pools.avail state=0x556addb801f0 s->prefetch_data=0 2021-02-11T00:29:29.007 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.004+0000 7f3fff610080 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-02-11T00:29:29.020 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.016+0000 7f3fff610080 10 WARNING: store->list_zones() returned r=-2 2021-02-11T00:29:29.020 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.016+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:29.020 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.016+0000 7f3fff610080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:29.024 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.020+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:29.024 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.020+0000 7f3fff610080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:29.024 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.020+0000 7f3fff610080 10 could not read realm id: (2) No such file or directory 2021-02-11T00:29:29.025 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.020+0000 7f3fff610080 10 WARNING: failed to set zone as default, r=-22 2021-02-11T00:29:29.025 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.020+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:29.025 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.020+0000 7f3fff610080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:29.029 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.024+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:29.029 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.024+0000 7f3fff610080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T00:29:29.030 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.024+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:29.030 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.024+0000 7f3fff610080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T00:29:29.030 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.024+0000 7f3fff610080 20 zonegroup default 2021-02-11T00:29:29.030 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.024+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:29.031 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.028+0000 7f3fff610080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:29.031 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.028+0000 7f3fff610080 10 Cannot find current period zone using local zone 2021-02-11T00:29:29.031 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.028+0000 7f3fff610080 10 Using default name default 2021-02-11T00:29:29.031 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.028+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:29.032 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.028+0000 7f3fff610080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:29.032 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.028+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:29.032 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.028+0000 7f3fff610080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T00:29:29.032 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.028+0000 7f3fff610080 20 zone default found 2021-02-11T00:29:29.032 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:29.028+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:30.999 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:30.992+0000 7f3fff610080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:30.999 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:30.992+0000 7f3fff610080 20 rados->read ofs=0 len=0 2021-02-11T00:29:30.999 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:30.996+0000 7f3fff610080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:30.999 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:30.996+0000 7f3fff610080 20 started sync module instance, tier type = 2021-02-11T00:29:30.999 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:30.996+0000 7f3fff610080 20 started zone id=dde3d945-4e16-429a-90a2-da7cefd73dc0 (name=default) with tier type = 2021-02-11T00:29:33.043 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:33.040+0000 7f3fff610080 20 add_watcher() i=0 2021-02-11T00:29:33.044 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:33.040+0000 7f3fff610080 20 add_watcher() i=1 2021-02-11T00:29:33.044 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:33.040+0000 7f3fff610080 20 add_watcher() i=2 2021-02-11T00:29:33.044 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:33.040+0000 7f3fff610080 20 add_watcher() i=3 2021-02-11T00:29:33.044 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:33.040+0000 7f3fff610080 20 add_watcher() i=4 2021-02-11T00:29:33.044 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:33.040+0000 7f3fff610080 20 add_watcher() i=5 2021-02-11T00:29:33.044 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:33.040+0000 7f3fff610080 20 add_watcher() i=6 2021-02-11T00:29:33.046 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:33.040+0000 7f3fff610080 20 add_watcher() i=7 2021-02-11T00:29:33.046 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:33.040+0000 7f3fff610080 2 all 8 watchers are set, enabling cache 2021-02-11T00:29:33.046 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:33.044+0000 7f3fff610080 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(optional_yield):423: oid=data_loggenerations_metadata not found 2021-02-11T00:29:33.060 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:33.056+0000 7f3fbbfef700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T00:29:33.060 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:33.056+0000 7f3fff610080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T00:29:33.060 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:33.056+0000 7f3fff610080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T00:29:37.020 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.016+0000 7f3fff610080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-02-11T00:29:37.154 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.148+0000 7f3f2affd700 20 reqs_thread_entry: start 2021-02-11T00:29:37.154 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.148+0000 7f3fff610080 20 init_complete bucket index max shards: 11 2021-02-11T00:29:37.154 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.148+0000 7f3f28ff9700 20 reqs_thread_entry: start 2021-02-11T00:29:37.154 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.148+0000 7f3fff610080 10 rgw notify: Started notification manager with: 1 workers 2021-02-11T00:29:37.154 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.148+0000 7f3f0effd700 20 rgw notify: INFO: next queues processing will happen at: Thu Feb 11 00:30:07 2021 2021-02-11T00:29:37.154 INFO:teuthology.orchestra.run.smithi095.stderr: 2021-02-11T00:29:37.154 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.148+0000 7f3f0e7fc700 20 reqs_thread_entry: start 2021-02-11T00:29:37.182 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.176+0000 7f3fff610080 20 remove_watcher() i=0 2021-02-11T00:29:37.183 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.176+0000 7f3fff610080 2 removed watcher, disabling cache 2021-02-11T00:29:37.185 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.180+0000 7f3fff610080 20 remove_watcher() i=1 2021-02-11T00:29:37.188 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.184+0000 7f3fff610080 20 remove_watcher() i=2 2021-02-11T00:29:37.193 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.188+0000 7f3fff610080 20 remove_watcher() i=3 2021-02-11T00:29:37.195 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.192+0000 7f3fff610080 20 remove_watcher() i=4 2021-02-11T00:29:37.198 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.192+0000 7f3fff610080 20 remove_watcher() i=5 2021-02-11T00:29:37.201 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.196+0000 7f3fff610080 20 remove_watcher() i=6 2021-02-11T00:29:37.206 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.200+0000 7f3fff610080 20 remove_watcher() i=7 2021-02-11T00:29:37.208 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.204+0000 7f3fbbfef700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T00:29:37.218 INFO:teuthology.orchestra.run.smithi095.stdout:[] 2021-02-11T00:29:37.218 DEBUG:tasks.util.rgw: json result: [] 2021-02-11T00:29:37.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T00:29:37.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T00:29:37.219 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-11T00:29:37.300 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:29:37.300 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:29:37.314 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.308+0000 7f7ab37b0080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T00:29:37.315 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.308+0000 7f7ab37b0080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T00:29:37.320 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.316+0000 7f7a38ff9700 20 reqs_thread_entry: start 2021-02-11T00:29:37.321 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.316+0000 7f7ab37b0080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.323 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.320+0000 7f7ab37b0080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:37.323 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.320+0000 7f7ab37b0080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.324 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.320+0000 7f7ab37b0080 20 rados_obj.operate() r=0 bl.length=873 2021-02-11T00:29:37.325 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.320+0000 7f7ab37b0080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.325 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.320+0000 7f7ab37b0080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:37.325 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.320+0000 7f7ab37b0080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.326 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.320+0000 7f7ab37b0080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:37.326 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.320+0000 7f7ab37b0080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.327 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.324+0000 7f7ab37b0080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T00:29:37.336 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"dde3d945-4e16-429a-90a2-da7cefd73dc0","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2021-02-11T00:29:37.337 DEBUG:tasks.util.rgw: json result: {'id': 'dde3d945-4e16-429a-90a2-da7cefd73dc0', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-02-11T00:29:37.337 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2021-02-11T00:29:37.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2021-02-11T00:29:37.338 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2021-02-11T00:29:37.414 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:29:37.414 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:29:37.424 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.420+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.426 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.420+0000 7fadec3a2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:37.426 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.420+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.427 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.424+0000 7fadec3a2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:37.427 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.424+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.427 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.424+0000 7fadec3a2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:37.427 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.424+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.428 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.424+0000 7fadec3a2080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:37.456 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.452+0000 7fadec3a2080 20 RGWRados::pool_iterate: got zone_info.dde3d945-4e16-429a-90a2-da7cefd73dc0 2021-02-11T00:29:37.456 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.452+0000 7fadec3a2080 20 RGWRados::pool_iterate: got zonegroup_info.40fb121c-1bbd-476a-ab3d-c7cc71da2d5f 2021-02-11T00:29:37.456 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.452+0000 7fadec3a2080 20 RGWRados::pool_iterate: got zone_names.default 2021-02-11T00:29:37.456 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.452+0000 7fadec3a2080 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-02-11T00:29:37.456 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.452+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.457 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.452+0000 7fadec3a2080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:37.457 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.452+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.458 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.452+0000 7fadec3a2080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T00:29:37.458 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.452+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.458 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.456+0000 7fadec3a2080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:37.459 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.456+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.459 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.456+0000 7fadec3a2080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T00:29:37.460 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.456+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.460 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.456+0000 7fadec3a2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:37.460 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.456+0000 7fadec3a2080 10 cannot find current period zonegroup using local zonegroup 2021-02-11T00:29:37.461 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.456+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.461 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.456+0000 7fadec3a2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:37.461 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.456+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.462 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.456+0000 7fadec3a2080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:37.462 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.456+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.463 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.460+0000 7fadec3a2080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T00:29:37.463 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.460+0000 7fadec3a2080 20 zonegroup default 2021-02-11T00:29:37.463 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.460+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.463 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.460+0000 7fadec3a2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:37.464 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.460+0000 7fadec3a2080 10 Cannot find current period zone using local zone 2021-02-11T00:29:37.464 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.460+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.464 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.460+0000 7fadec3a2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:37.464 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.460+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.465 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.460+0000 7fadec3a2080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:37.465 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.460+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.466 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.460+0000 7fadec3a2080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T00:29:37.466 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.460+0000 7fadec3a2080 20 zone default found 2021-02-11T00:29:37.466 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.460+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.467 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.464+0000 7fadec3a2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:37.467 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.464+0000 7fadec3a2080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.467 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.464+0000 7fadec3a2080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:37.467 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.464+0000 7fadec3a2080 20 started sync module instance, tier type = 2021-02-11T00:29:37.468 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.464+0000 7fadec3a2080 20 started zone id=dde3d945-4e16-429a-90a2-da7cefd73dc0 (name=default) with tier type = 2021-02-11T00:29:37.491 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.488+0000 7fadec3a2080 20 add_watcher() i=0 2021-02-11T00:29:37.491 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.488+0000 7fadec3a2080 20 add_watcher() i=1 2021-02-11T00:29:37.491 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.488+0000 7fadec3a2080 20 add_watcher() i=2 2021-02-11T00:29:37.491 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.488+0000 7fadec3a2080 20 add_watcher() i=3 2021-02-11T00:29:37.491 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.488+0000 7fadec3a2080 20 add_watcher() i=4 2021-02-11T00:29:37.491 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.488+0000 7fadec3a2080 20 add_watcher() i=5 2021-02-11T00:29:37.491 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.488+0000 7fadec3a2080 20 add_watcher() i=6 2021-02-11T00:29:37.493 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.488+0000 7fadec3a2080 20 add_watcher() i=7 2021-02-11T00:29:37.493 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.488+0000 7fadec3a2080 2 all 8 watchers are set, enabling cache 2021-02-11T00:29:37.496 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.492+0000 7fada97e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T00:29:37.496 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.492+0000 7fadec3a2080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T00:29:37.496 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.492+0000 7fadec3a2080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T00:29:37.580 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.576+0000 7fad17fff700 20 reqs_thread_entry: start 2021-02-11T00:29:37.580 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.576+0000 7fad15ffb700 20 reqs_thread_entry: start 2021-02-11T00:29:37.580 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.576+0000 7fadec3a2080 20 init_complete bucket index max shards: 11 2021-02-11T00:29:37.581 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.576+0000 7fadec3a2080 10 rgw notify: Started notification manager with: 1 workers 2021-02-11T00:29:37.581 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.576+0000 7fad0bfff700 20 rgw notify: INFO: next queues processing will happen at: Thu Feb 11 00:30:07 2021 2021-02-11T00:29:37.581 INFO:teuthology.orchestra.run.smithi095.stderr: 2021-02-11T00:29:37.581 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.576+0000 7fad0b7fe700 20 reqs_thread_entry: start 2021-02-11T00:29:37.608 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.604+0000 7fadec3a2080 20 remove_watcher() i=0 2021-02-11T00:29:37.608 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.604+0000 7fadec3a2080 2 removed watcher, disabling cache 2021-02-11T00:29:37.611 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.608+0000 7fadec3a2080 20 remove_watcher() i=1 2021-02-11T00:29:37.614 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.608+0000 7fadec3a2080 20 remove_watcher() i=2 2021-02-11T00:29:37.616 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.612+0000 7fadec3a2080 20 remove_watcher() i=3 2021-02-11T00:29:37.618 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.616+0000 7fadec3a2080 20 remove_watcher() i=4 2021-02-11T00:29:37.621 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.616+0000 7fadec3a2080 20 remove_watcher() i=5 2021-02-11T00:29:37.623 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.620+0000 7fadec3a2080 20 remove_watcher() i=6 2021-02-11T00:29:37.626 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.620+0000 7fadec3a2080 20 remove_watcher() i=7 2021-02-11T00:29:37.628 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.624+0000 7fada97e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-02-11T00:29:37.636 INFO:teuthology.orchestra.run.smithi095.stdout:[] 2021-02-11T00:29:37.636 DEBUG:tasks.util.rgw: json result: [] 2021-02-11T00:29:37.636 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T00:29:37.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-02-11T00:29:37.637 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-02-11T00:29:37.717 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:29:37.718 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:29:37.728 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.724+0000 7fb140b41080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-02-11T00:29:37.728 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.724+0000 7fb140b41080 20 check_secure_mon_conn(): mode 1 is insecure 2021-02-11T00:29:37.736 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.732+0000 7fb140b41080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.736 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.732+0000 7fb0a57fa700 20 reqs_thread_entry: start 2021-02-11T00:29:37.738 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.732+0000 7fb140b41080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:37.738 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.732+0000 7fb140b41080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.739 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.736+0000 7fb140b41080 20 rados_obj.operate() r=0 bl.length=889 2021-02-11T00:29:37.740 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.736+0000 7fb140b41080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.740 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.736+0000 7fb140b41080 20 rados_obj.operate() r=-2 bl.length=0 2021-02-11T00:29:37.740 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.736+0000 7fb140b41080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.741 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.736+0000 7fb140b41080 20 rados_obj.operate() r=0 bl.length=46 2021-02-11T00:29:37.741 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.736+0000 7fb140b41080 20 rados->read ofs=0 len=0 2021-02-11T00:29:37.741 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:37.736+0000 7fb140b41080 20 rados_obj.operate() r=0 bl.length=358 2021-02-11T00:29:37.750 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"dde3d945-4e16-429a-90a2-da7cefd73dc0","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2021-02-11T00:29:37.751 DEBUG:tasks.util.rgw: json result: {'id': 'dde3d945-4e16-429a-90a2-da7cefd73dc0', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-02-11T00:29:37.751 INFO:tasks.rgw:Starting rgw... 2021-02-11T00:29:37.751 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2021-02-11T00:29:37.751 INFO:tasks.rgw:Using beast as radosgw frontend 2021-02-11T00:29:37.751 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-02-11T00:29:37.751 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-02-11T00:29:37.754 INFO:tasks.rgw.c1.client.0:Started 2021-02-11T00:29:37.754 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2021-02-11T00:29:37.754 INFO:tasks.rgw:Using beast as radosgw frontend 2021-02-11T00:29:37.754 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-02-11T00:29:37.754 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-02-11T00:29:37.756 INFO:tasks.rgw.c1.client.1:Started 2021-02-11T00:29:37.756 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2021-02-11T00:29:37.757 INFO:tasks.rgw:Using beast as radosgw frontend 2021-02-11T00:29:37.757 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-02-11T00:29:37.757 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-02-11T00:29:37.795 INFO:tasks.rgw.c2.client.0:Started 2021-02-11T00:29:37.796 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2021-02-11T00:29:37.796 INFO:tasks.rgw:Using beast as radosgw frontend 2021-02-11T00:29:37.796 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-02-11T00:29:37.796 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-02-11T00:29:37.798 INFO:tasks.rgw.c2.client.1:Started 2021-02-11T00:29:37.798 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi057.front.sepia.ceph.com:8000/ 2021-02-11T00:29:37.798 DEBUG:teuthology.orchestra.run.smithi057:> curl http://smithi057.front.sepia.ceph.com:8000/ 2021-02-11T00:29:37.826 INFO:teuthology.orchestra.run.smithi057.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T00:29:37.826 INFO:teuthology.orchestra.run.smithi057.stderr: Dload Upload Total Spent Left Speed 2021-02-11T00:29:37.828 INFO:teuthology.orchestra.run.smithi057.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi057.front.sepia.ceph.com port 8000: Connection refused 2021-02-11T00:29:37.828 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T00:29:38.829 DEBUG:teuthology.orchestra.run.smithi057:> curl http://smithi057.front.sepia.ceph.com:8000/ 2021-02-11T00:29:38.847 INFO:teuthology.orchestra.run.smithi057.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T00:29:38.847 INFO:teuthology.orchestra.run.smithi057.stderr: Dload Upload Total Spent Left Speed 2021-02-11T00:29:38.855 INFO:teuthology.orchestra.run.smithi057.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 23777 0 --:--:-- --:--:-- --:--:-- 26750 2021-02-11T00:29:38.856 INFO:teuthology.orchestra.run.smithi057.stdout:anonymous 2021-02-11T00:29:38.857 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi057.front.sepia.ceph.com:8001/ 2021-02-11T00:29:38.857 DEBUG:teuthology.orchestra.run.smithi057:> curl http://smithi057.front.sepia.ceph.com:8001/ 2021-02-11T00:29:38.911 INFO:teuthology.orchestra.run.smithi057.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T00:29:38.911 INFO:teuthology.orchestra.run.smithi057.stderr: Dload Upload Total Spent Left Speed 2021-02-11T00:29:38.918 INFO:teuthology.orchestra.run.smithi057.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2021-02-11T00:29:38.918 INFO:teuthology.orchestra.run.smithi057.stdout:anonymous 2021-02-11T00:29:38.919 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi095.front.sepia.ceph.com:8000/ 2021-02-11T00:29:38.919 DEBUG:teuthology.orchestra.run.smithi095:> curl http://smithi095.front.sepia.ceph.com:8000/ 2021-02-11T00:29:39.009 INFO:teuthology.orchestra.run.smithi095.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T00:29:39.009 INFO:teuthology.orchestra.run.smithi095.stderr: Dload Upload Total Spent Left Speed 2021-02-11T00:29:39.016 INFO:teuthology.orchestra.run.smithi095.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 14266 0 --:--:-- --:--:-- --:--:-- 30571 2021-02-11T00:29:39.016 INFO:teuthology.orchestra.run.smithi095.stdout:anonymous 2021-02-11T00:29:39.017 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi095.front.sepia.ceph.com:8001/ 2021-02-11T00:29:39.017 DEBUG:teuthology.orchestra.run.smithi095:> curl http://smithi095.front.sepia.ceph.com:8001/ 2021-02-11T00:29:39.031 INFO:teuthology.orchestra.run.smithi095.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T00:29:39.031 INFO:teuthology.orchestra.run.smithi095.stderr: Dload Upload Total Spent Left Speed 2021-02-11T00:29:39.038 INFO:teuthology.orchestra.run.smithi095.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2021-02-11T00:29:39.038 INFO:teuthology.orchestra.run.smithi095.stdout:anonymous 2021-02-11T00:29:39.039 INFO:teuthology.run_tasks:Running task rgw-multisite... 2021-02-11T00:29:39.536 INFO:tasks.rgw_multisite:creating realm.. 2021-02-11T00:29:39.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:29:39.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:29:39.536 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:29:39.575 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:29:39.575 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:29:39.647 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"9c192af4-f67d-481a-af12-4cc9f791da82","name":"test-realm","current_period":"26b3dab9-eb6f-4146-98fb-6058d0d1ab68","epoch":1} 2021-02-11T00:29:39.648 INFO:tasks.rgw_multisite:creating master zonegroup.. 2021-02-11T00:29:39.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:29:39.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:29:39.648 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:29:39.730 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:29:39.730 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:29:39.755 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:29:39.749+0000 7fd72fc78080 -1 old format 2021-02-11T00:29:39.788 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}} 2021-02-11T00:29:39.789 INFO:tasks.rgw_multisite:creating master zone.. 2021-02-11T00:29:39.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--endpoints', 'http://smithi057.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:29:39.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--endpoints', 'http://smithi057.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:29:39.789 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key CP2FO0ORQU6DSQKV --secret szUDueomrwAKvai4rUGz9pCzuDdncCWD --endpoints http://smithi057.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:29:39.878 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:29:39.878 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:29:39.973 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"CP2FO0ORQU6DSQKV","secret_key":"szUDueomrwAKvai4rUGz9pCzuDdncCWD"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","notif_pool":"test-zone1.rgw.log:notif"} 2021-02-11T00:29:39.975 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2021-02-11T00:29:40.828 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'test-zone1.rgw.buckets.data' created 2021-02-11T00:29:40.852 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2021-02-11T00:29:41.854 INFO:teuthology.orchestra.run.smithi057.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2021-02-11T00:29:41.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:29:41.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:29:41.875 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:29:41.922 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:29:41.922 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:29:41.947 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"CP2FO0ORQU6DSQKV","secret_key":"szUDueomrwAKvai4rUGz9pCzuDdncCWD"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"test-zone1.rgw.log:notif"} 2021-02-11T00:29:41.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:29:41.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:29:41.948 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:29:41.986 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:29:41.986 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:29:50.160 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"7f3547bf-73e3-43a7-aa91-648db0f982f6","epoch":1,"predecessor_uuid":"26b3dab9-eb6f-4146-98fb-6058d0d1ab68","sync_status":[],"period_map":{"id":"7f3547bf-73e3-43a7-aa91-648db0f982f6","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","zones":[{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","val":3133685257}]},"master_zonegroup":"test-zonegroup","master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","realm_name":"test-realm","realm_epoch":2} 2021-02-11T00:29:50.161 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-02-11T00:29:50.161 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T00:29:50.208 INFO:tasks.rgw.c1.client.0.smithi057.stdout:2021-02-11T00:29:50.205+0000 7f95e77fe700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 17041) UID: 0 2021-02-11T00:29:50.208 INFO:tasks.rgw.c1.client.0.smithi057.stdout:2021-02-11T00:29:50.205+0000 7f960b74d0c0 -1 shutting down 2021-02-11T00:29:53.109 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-02-11T00:29:56.262 INFO:tasks.rgw.c1.client.0:Stopped 2021-02-11T00:29:56.263 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-02-11T00:29:56.263 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-02-11T00:29:56.265 INFO:tasks.rgw.c1.client.0:Started 2021-02-11T00:29:56.265 DEBUG:teuthology.orchestra.run.smithi057:> curl http://smithi057.front.sepia.ceph.com:8000 2021-02-11T00:29:56.282 INFO:teuthology.orchestra.run.smithi057.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T00:29:56.282 INFO:teuthology.orchestra.run.smithi057.stderr: Dload Upload Total Spent Left Speed 2021-02-11T00:29:56.286 INFO:teuthology.orchestra.run.smithi057.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi057.front.sepia.ceph.com port 8000: Connection refused 2021-02-11T00:29:56.287 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T00:29:57.288 DEBUG:teuthology.orchestra.run.smithi057:> curl http://smithi057.front.sepia.ceph.com:8000 2021-02-11T00:29:57.305 INFO:teuthology.orchestra.run.smithi057.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T00:29:57.305 INFO:teuthology.orchestra.run.smithi057.stderr: Dload Upload Total Spent Left Speed 2021-02-11T00:29:57.311 INFO:teuthology.orchestra.run.smithi057.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 35666 2021-02-11T00:29:57.312 INFO:teuthology.orchestra.run.smithi057.stdout:anonymous 2021-02-11T00:29:57.312 INFO:tasks.rgw_multisite:creating admin user.. 2021-02-11T00:29:57.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:29:57.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:29:57.313 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key CP2FO0ORQU6DSQKV --secret szUDueomrwAKvai4rUGz9pCzuDdncCWD --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:29:57.401 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:29:57.401 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:29:57.605 INFO:teuthology.orchestra.run.smithi057.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"CP2FO0ORQU6DSQKV","secret_key":"szUDueomrwAKvai4rUGz9pCzuDdncCWD"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":"true","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2021-02-11T00:29:57.637 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-02-11T00:29:57.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi057.front.sepia.ceph.com:8000', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:29:57.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi057.front.sepia.ceph.com:8000', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:29:57.637 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi057.front.sepia.ceph.com:8000 --access-key CP2FO0ORQU6DSQKV --secret szUDueomrwAKvai4rUGz9pCzuDdncCWD --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:29:57.678 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:29:57.678 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:29:57.799 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"9c192af4-f67d-481a-af12-4cc9f791da82","name":"test-realm","current_period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","epoch":2} 2021-02-11T00:29:57.799 INFO:tasks.rgw_multisite:creating zone.. 2021-02-11T00:29:57.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--endpoints', 'http://smithi095.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:29:57.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--endpoints', 'http://smithi095.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:29:57.800 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key CP2FO0ORQU6DSQKV --secret szUDueomrwAKvai4rUGz9pCzuDdncCWD --endpoints http://smithi095.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:29:57.890 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:29:57.890 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:29:57.944 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:57.940+0000 7fa3935df080 0 failed reading obj info from .rgw.root:zone_info.e989c9f2-9c89-43f5-a54f-e1d0705a9abe: (2) No such file or directory 2021-02-11T00:29:57.944 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:29:57.940+0000 7fa3935df080 0 WARNING: could not read zone params for zone id=e989c9f2-9c89-43f5-a54f-e1d0705a9abe name=test-zone1 2021-02-11T00:29:57.961 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"CP2FO0ORQU6DSQKV","secret_key":"szUDueomrwAKvai4rUGz9pCzuDdncCWD"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T00:29:57.962 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2021-02-11T00:29:58.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-02-11T00:29:59.009 INFO:teuthology.orchestra.run.smithi095.stderr:pool 'test-zone2.rgw.buckets.data' created 2021-02-11T00:29:59.026 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2021-02-11T00:30:00.055 INFO:teuthology.orchestra.run.smithi095.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2021-02-11T00:30:00.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:00.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:00.073 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:30:00.122 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:30:00.122 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:30:00.149 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"CP2FO0ORQU6DSQKV","secret_key":"szUDueomrwAKvai4rUGz9pCzuDdncCWD"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T00:30:00.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:00.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:00.149 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:30:00.228 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:30:00.228 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:30:08.219 INFO:teuthology.orchestra.run.smithi095.stderr:Sending period to new master zone e989c9f2-9c89-43f5-a54f-e1d0705a9abe 2021-02-11T00:30:08.334 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"7f3547bf-73e3-43a7-aa91-648db0f982f6","epoch":2,"predecessor_uuid":"26b3dab9-eb6f-4146-98fb-6058d0d1ab68","sync_status":[],"period_map":{"id":"7f3547bf-73e3-43a7-aa91-648db0f982f6","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","zones":[{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","endpoints":["http://smithi095.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","val":3133685257},{"key":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","val":1241439546}]},"master_zonegroup":"test-zonegroup","master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","realm_name":"test-realm","realm_epoch":2} 2021-02-11T00:30:08.334 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-02-11T00:30:08.334 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T00:30:08.379 INFO:tasks.rgw.c2.client.0.smithi095.stdout:2021-02-11T00:30:08.376+0000 7fafe61aa700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 17005) UID: 0 2021-02-11T00:30:08.379 INFO:tasks.rgw.c2.client.0.smithi095.stdout:2021-02-11T00:30:08.376+0000 7faffdd050c0 -1 shutting down 2021-02-11T00:30:10.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-02-11T00:30:14.437 INFO:tasks.rgw.c2.client.0:Stopped 2021-02-11T00:30:14.437 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-02-11T00:30:14.437 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-02-11T00:30:14.440 INFO:tasks.rgw.c2.client.0:Started 2021-02-11T00:30:14.440 DEBUG:teuthology.orchestra.run.smithi095:> curl http://smithi095.front.sepia.ceph.com:8000 2021-02-11T00:30:14.457 INFO:teuthology.orchestra.run.smithi095.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T00:30:14.457 INFO:teuthology.orchestra.run.smithi095.stderr: Dload Upload Total Spent Left Speed 2021-02-11T00:30:14.460 INFO:teuthology.orchestra.run.smithi095.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi095.front.sepia.ceph.com port 8000: Connection refused 2021-02-11T00:30:14.461 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T00:30:15.462 DEBUG:teuthology.orchestra.run.smithi095:> curl http://smithi095.front.sepia.ceph.com:8000 2021-02-11T00:30:15.476 INFO:teuthology.orchestra.run.smithi095.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T00:30:15.476 INFO:teuthology.orchestra.run.smithi095.stderr: Dload Upload Total Spent Left Speed 2021-02-11T00:30:15.483 INFO:teuthology.orchestra.run.smithi095.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2021-02-11T00:30:15.484 INFO:teuthology.orchestra.run.smithi095.stdout:anonymous 2021-02-11T00:30:15.484 INFO:tasks.rgw_multisite:creating zone.. 2021-02-11T00:30:15.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--endpoints', 'http://smithi057.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:15.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--endpoints', 'http://smithi057.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:15.485 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key CP2FO0ORQU6DSQKV --secret szUDueomrwAKvai4rUGz9pCzuDdncCWD --endpoints http://smithi057.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:30:15.529 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:30:15.530 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:30:15.607 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:30:15.605+0000 7fa136196080 0 failed reading obj info from .rgw.root:zone_info.fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605: (2) No such file or directory 2021-02-11T00:30:15.607 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:30:15.605+0000 7fa136196080 0 WARNING: could not read zone params for zone id=fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605 name=test-zone2 2021-02-11T00:30:15.624 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"35e954f9-5ea0-411e-8e41-e821d2787b31","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"CP2FO0ORQU6DSQKV","secret_key":"szUDueomrwAKvai4rUGz9pCzuDdncCWD"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","notif_pool":"test-zone3.rgw.log:notif"} 2021-02-11T00:30:15.624 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2021-02-11T00:30:16.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2021-02-11T00:30:16.864 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'test-zone3.rgw.buckets.data' created 2021-02-11T00:30:16.887 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2021-02-11T00:30:17.899 INFO:teuthology.orchestra.run.smithi057.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2021-02-11T00:30:17.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:17.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:17.914 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:30:17.964 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:30:17.965 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:30:17.991 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"35e954f9-5ea0-411e-8e41-e821d2787b31","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"CP2FO0ORQU6DSQKV","secret_key":"szUDueomrwAKvai4rUGz9pCzuDdncCWD"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"test-zone3.rgw.log:notif"} 2021-02-11T00:30:17.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:17.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:17.992 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:30:18.027 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:30:18.028 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:30:26.091 INFO:teuthology.orchestra.run.smithi057.stderr:Sending period to new master zone e989c9f2-9c89-43f5-a54f-e1d0705a9abe 2021-02-11T00:30:26.189 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"7f3547bf-73e3-43a7-aa91-648db0f982f6","epoch":3,"predecessor_uuid":"26b3dab9-eb6f-4146-98fb-6058d0d1ab68","sync_status":[],"period_map":{"id":"7f3547bf-73e3-43a7-aa91-648db0f982f6","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","zones":[{"id":"35e954f9-5ea0-411e-8e41-e821d2787b31","name":"test-zone3","endpoints":["http://smithi057.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","endpoints":["http://smithi095.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"35e954f9-5ea0-411e-8e41-e821d2787b31","val":398828966},{"key":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","val":3133685257},{"key":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","val":1241439546}]},"master_zonegroup":"test-zonegroup","master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","realm_name":"test-realm","realm_epoch":2} 2021-02-11T00:30:26.190 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-02-11T00:30:26.190 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T00:30:26.231 INFO:tasks.rgw.c1.client.1.smithi057.stdout:2021-02-11T00:30:26.229+0000 7fe432d28700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 17042) UID: 0 2021-02-11T00:30:26.232 INFO:tasks.rgw.c1.client.1.smithi057.stdout:2021-02-11T00:30:26.229+0000 7fe44a8830c0 -1 shutting down 2021-02-11T00:30:27.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-02-11T00:30:32.291 INFO:tasks.rgw.c1.client.1:Stopped 2021-02-11T00:30:32.291 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-02-11T00:30:32.291 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-02-11T00:30:32.294 INFO:tasks.rgw.c1.client.1:Started 2021-02-11T00:30:32.294 DEBUG:teuthology.orchestra.run.smithi057:> curl http://smithi057.front.sepia.ceph.com:8001 2021-02-11T00:30:32.310 INFO:teuthology.orchestra.run.smithi057.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T00:30:32.310 INFO:teuthology.orchestra.run.smithi057.stderr: Dload Upload Total Spent Left Speed 2021-02-11T00:30:32.314 INFO:teuthology.orchestra.run.smithi057.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi057.front.sepia.ceph.com port 8001: Connection refused 2021-02-11T00:30:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T00:30:32.952 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-02-11T00:30:33.316 DEBUG:teuthology.orchestra.run.smithi057:> curl http://smithi057.front.sepia.ceph.com:8001 2021-02-11T00:30:33.331 INFO:teuthology.orchestra.run.smithi057.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T00:30:33.331 INFO:teuthology.orchestra.run.smithi057.stderr: Dload Upload Total Spent Left Speed 2021-02-11T00:30:33.338 INFO:teuthology.orchestra.run.smithi057.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2021-02-11T00:30:33.339 INFO:teuthology.orchestra.run.smithi057.stdout:anonymous 2021-02-11T00:30:33.339 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-02-11T00:30:33.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi057.front.sepia.ceph.com:8000', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--url', 'http://smithi057.front.sepia.ceph.com:8000', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:33.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi057.front.sepia.ceph.com:8000', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--url', 'http://smithi057.front.sepia.ceph.com:8000', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:33.340 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi057.front.sepia.ceph.com:8000 --access-key CP2FO0ORQU6DSQKV --secret szUDueomrwAKvai4rUGz9pCzuDdncCWD --url http://smithi057.front.sepia.ceph.com:8000 --access-key CP2FO0ORQU6DSQKV --secret szUDueomrwAKvai4rUGz9pCzuDdncCWD --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:30:33.390 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:30:33.390 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:30:33.438 INFO:tasks.rgw.c2.client.0.smithi095.stdout:2021-02-11T00:30:33.436+0000 7f682e5fc700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.9c192af4-f67d-481a-af12-4cc9f791da82.control err=-107 2021-02-11T00:30:33.462 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"9c192af4-f67d-481a-af12-4cc9f791da82","name":"test-realm","current_period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","epoch":2} 2021-02-11T00:30:33.463 INFO:tasks.rgw_multisite:creating zone.. 2021-02-11T00:30:33.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--endpoints', 'http://smithi095.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:33.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--endpoints', 'http://smithi095.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:33.463 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key CP2FO0ORQU6DSQKV --secret szUDueomrwAKvai4rUGz9pCzuDdncCWD --endpoints http://smithi095.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:30:33.515 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:30:33.515 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:30:33.579 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:30:33.576+0000 7f9c1f7ad080 0 failed reading obj info from .rgw.root:zone_info.35e954f9-5ea0-411e-8e41-e821d2787b31: (2) No such file or directory 2021-02-11T00:30:33.579 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:30:33.576+0000 7f9c1f7ad080 0 WARNING: could not read zone params for zone id=35e954f9-5ea0-411e-8e41-e821d2787b31 name=test-zone3 2021-02-11T00:30:33.580 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:30:33.576+0000 7f9c1f7ad080 0 failed reading obj info from .rgw.root:zone_info.e989c9f2-9c89-43f5-a54f-e1d0705a9abe: (2) No such file or directory 2021-02-11T00:30:33.580 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:30:33.576+0000 7f9c1f7ad080 0 WARNING: could not read zone params for zone id=e989c9f2-9c89-43f5-a54f-e1d0705a9abe name=test-zone1 2021-02-11T00:30:33.591 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"CP2FO0ORQU6DSQKV","secret_key":"szUDueomrwAKvai4rUGz9pCzuDdncCWD"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","notif_pool":"test-zone4.rgw.log:notif"} 2021-02-11T00:30:33.591 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2021-02-11T00:30:34.006 INFO:teuthology.orchestra.run.smithi095.stderr:pool 'test-zone4.rgw.buckets.data' created 2021-02-11T00:30:34.019 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2021-02-11T00:30:35.040 INFO:teuthology.orchestra.run.smithi095.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2021-02-11T00:30:35.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:35.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:35.060 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:30:35.103 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:30:35.103 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:30:35.130 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"CP2FO0ORQU6DSQKV","secret_key":"szUDueomrwAKvai4rUGz9pCzuDdncCWD"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"","notif_pool":"test-zone4.rgw.log:notif"} 2021-02-11T00:30:35.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:35.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:35.131 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:30:35.212 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:30:35.212 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:30:43.247 INFO:teuthology.orchestra.run.smithi095.stderr:Sending period to new master zone e989c9f2-9c89-43f5-a54f-e1d0705a9abe 2021-02-11T00:30:43.349 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"7f3547bf-73e3-43a7-aa91-648db0f982f6","epoch":4,"predecessor_uuid":"26b3dab9-eb6f-4146-98fb-6058d0d1ab68","sync_status":[],"period_map":{"id":"7f3547bf-73e3-43a7-aa91-648db0f982f6","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","zones":[{"id":"35e954f9-5ea0-411e-8e41-e821d2787b31","name":"test-zone3","endpoints":["http://smithi057.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","name":"test-zone4","endpoints":["http://smithi095.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","endpoints":["http://smithi095.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"35e954f9-5ea0-411e-8e41-e821d2787b31","val":398828966},{"key":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","val":1248431312},{"key":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","val":3133685257},{"key":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","val":1241439546}]},"master_zonegroup":"test-zonegroup","master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","realm_name":"test-realm","realm_epoch":2} 2021-02-11T00:30:43.349 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-02-11T00:30:43.349 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T00:30:43.390 INFO:tasks.rgw.c2.client.1.smithi095.stdout:2021-02-11T00:30:43.388+0000 7f63feffd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 17006) UID: 0 2021-02-11T00:30:43.390 INFO:tasks.rgw.c2.client.1.smithi095.stdout:2021-02-11T00:30:43.388+0000 7f64230170c0 -1 shutting down 2021-02-11T00:30:44.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-02-11T00:30:49.451 INFO:tasks.rgw.c2.client.1:Stopped 2021-02-11T00:30:49.451 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-02-11T00:30:49.451 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-02-11T00:30:49.455 INFO:tasks.rgw.c2.client.1:Started 2021-02-11T00:30:49.455 DEBUG:teuthology.orchestra.run.smithi095:> curl http://smithi095.front.sepia.ceph.com:8001 2021-02-11T00:30:49.470 INFO:teuthology.orchestra.run.smithi095.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T00:30:49.471 INFO:teuthology.orchestra.run.smithi095.stderr: Dload Upload Total Spent Left Speed 2021-02-11T00:30:49.474 INFO:teuthology.orchestra.run.smithi095.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi095.front.sepia.ceph.com port 8001: Connection refused 2021-02-11T00:30:49.475 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T00:30:50.476 DEBUG:teuthology.orchestra.run.smithi095:> curl http://smithi095.front.sepia.ceph.com:8001 2021-02-11T00:30:50.490 INFO:teuthology.orchestra.run.smithi095.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T00:30:50.490 INFO:teuthology.orchestra.run.smithi095.stderr: Dload Upload Total Spent Left Speed 2021-02-11T00:30:50.495 INFO:teuthology.orchestra.run.smithi095.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 26800 0 --:--:-- --:--:-- --:--:-- 33500 2021-02-11T00:30:50.496 INFO:teuthology.orchestra.run.smithi095.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-0060247aba-10e7-test-zone4","HostId":"10e7-test-zone4-test-zonegroup"} 2021-02-11T00:30:50.496 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2021-02-11T00:30:50.496 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2021-02-11T00:30:50.631 INFO:tasks.rgw_multisite_tests:creating test user.. 2021-02-11T00:30:50.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:50.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:30:50.632 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:30:50.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2021-02-11T00:30:50.678 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:30:50.678 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:30:50.868 INFO:teuthology.orchestra.run.smithi057.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"13G61HO8RD7VIB5MPORO","secret_key":"TY2UnwbR6G5AIItRXHKFm8YEGBnErcKfkz6rVxXW"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2021-02-11T00:30:50.902 INFO:rgw_multi.tests:meta checkpoint 2021-02-11T00:30:50.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:30:50.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:30:50.902 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:30:50.985 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:30:50.985 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:30:51.212 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:30:51.249 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003450.863150_45.1","last_update":"2021-02-11T00:30:50.863150Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:30:51.249 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:30:51.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:30:51.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:30:51.250 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:30:51.291 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:30:51.291 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:30:51.520 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:30:51.520 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:30:56.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:30:56.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:30:56.522 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:30:56.577 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:30:56.577 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:30:57.144 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:30:57.144 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:31:02.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:02.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:02.146 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:31:02.187 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:31:02.187 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:31:02.415 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:31:02.416 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:31:07.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:07.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:07.418 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:31:07.460 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:31:07.460 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:31:07.689 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:31:07.689 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:31:12.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:12.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:12.691 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:31:12.731 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:31:12.731 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:31:12.947 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:31:12.947 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:31:17.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:17.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:17.949 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:31:17.989 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:31:17.989 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:31:18.216 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:31:18.217 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:31:23.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:23.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:23.219 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:31:23.258 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:31:23.259 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:31:23.482 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:31:23.483 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:31:28.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:28.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:28.485 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:31:28.526 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:31:28.526 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:31:28.758 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:31:28.759 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:31:33.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:33.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:33.760 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:31:33.802 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:31:33.803 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:31:34.033 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:31:34.033 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:31:39.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:39.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:39.035 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:31:39.075 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:31:39.076 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:31:39.315 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:31:39.315 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:31:44.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:44.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:44.317 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:31:44.360 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:31:44.360 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:31:44.596 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:31:44.596 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:31:49.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:49.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:49.597 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:31:49.639 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:31:49.639 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:31:49.866 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:31:49.867 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:31:54.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:54.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:31:54.868 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:31:54.909 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:31:54.909 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:31:55.126 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:31:55.126 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:32:00.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:00.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:00.127 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:00.167 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:00.167 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:00.382 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:00.383 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:32:05.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:05.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:05.384 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:05.425 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:05.425 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:05.657 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:05.657 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:32:10.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:10.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:10.659 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:10.700 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:10.700 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:10.939 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:10.939 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:32:15.941 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:15.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:15.941 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:15.982 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:15.982 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:16.220 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:16.220 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:32:21.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:22.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:22.431 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:22.472 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:22.472 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:22.708 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:22.708 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:32:27.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:27.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:27.710 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:27.751 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:27.751 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:27.981 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:27.981 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:32:32.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:32.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:32.983 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:33.025 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:33.025 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:33.265 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:33.265 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:32:38.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:38.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:38.267 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:38.310 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:38.310 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:38.547 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:38.547 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:32:43.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:43.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:43.549 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:43.592 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:43.592 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:43.833 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:43.834 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1613003450.863150_45.1 target= 2021-02-11T00:32:48.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:48.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:48.835 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:48.878 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:48.878 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:49.097 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003450.863150_45.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:30:50.863150Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:49.098 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:32:49.098 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:32:49.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:49.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:49.098 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:49.139 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:49.139 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:49.391 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003450.863150_45.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:30:50.863150Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:49.391 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:32:49.392 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:32:49.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:49.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:49.392 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:49.433 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:49.433 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:49.668 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003450.863150_45.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:30:50.863150Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:49.669 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:32:49.669 INFO:rgw_multi.tests:meta checkpoint 2021-02-11T00:32:49.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:49.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:49.669 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:49.709 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:49.709 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:49.904 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:32:49.940 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003450.863150_45.1","last_update":"2021-02-11T00:30:50.863150Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:32:49.941 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:32:49.941 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:49.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:49.941 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:49.982 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:49.983 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:50.201 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003450.863150_45.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:30:50.863150Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:50.201 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:32:50.201 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:32:50.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:50.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:50.202 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:50.242 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:50.242 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:50.478 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003450.863150_45.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:30:50.863150Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:50.478 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:32:50.478 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:32:50.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:50.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:50.479 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:50.520 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:50.520 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:50.746 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003450.863150_45.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:30:50.863150Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:50.746 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:32:50.746 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2021-02-11T00:32:50.759 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-1 2021-02-11T00:32:53.943 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-2 2021-02-11T00:32:57.079 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-3 2021-02-11T00:32:57.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:57.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:57.169 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:57.204 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:57.204 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:57.402 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:32:57.433 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003577.148691_59.1","last_update":"2021-02-11T00:32:57.148691Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:32:57.434 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:32:57.434 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:57.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:57.434 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:57.470 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:57.470 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:57.708 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003577.148691_59.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:32:57.148691Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:57.708 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:32:57.708 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:32:57.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:57.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:57.709 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:57.750 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:57.750 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:58.020 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003577.148691_59.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:32:57.148691Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:58.021 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:32:58.021 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:32:58.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:58.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:58.021 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:58.073 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:58.073 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:58.291 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003577.148691_59.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:32:57.148691Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:58.291 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:32:58.333 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2021-02-11T00:32:58.334 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-4 2021-02-11T00:32:58.363 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-5 2021-02-11T00:32:58.415 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-6 2021-02-11T00:32:58.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:58.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:58.465 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:58.499 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:58.499 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:58.687 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:32:58.725 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003578.448531_71.1","last_update":"2021-02-11T00:32:58.448531Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:32:58.726 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:32:58.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:58.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:58.726 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:58.765 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:58.765 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:59.014 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003578.448531_71.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:32:58.448531Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:59.014 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:32:59.014 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:32:59.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:59.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:59.015 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:59.057 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:59.057 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:59.282 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003578.448531_71.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:32:58.448531Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:59.282 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:32:59.283 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:32:59.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:59.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:59.283 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:59.325 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:59.325 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:32:59.544 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003578.448531_71.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:32:58.448531Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:32:59.544 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:32:59.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:59.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:32:59.887 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:32:59.922 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:32:59.923 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:00.109 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:33:00.147 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003578.448531_71.1","last_update":"2021-02-11T00:32:58.448531Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:33:00.147 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:33:00.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:00.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:00.148 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:00.189 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:00.190 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:00.418 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003578.448531_71.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:32:58.448531Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:00.418 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:33:00.418 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:33:00.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:00.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:00.419 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:00.459 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:00.459 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:00.710 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003578.448531_71.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:32:58.448531Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:00.711 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:33:00.711 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:33:00.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:00.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:00.711 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:00.752 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:00.752 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:00.971 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003578.448531_71.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:32:58.448531Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:00.972 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:33:01.012 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2021-02-11T00:33:01.013 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-7 2021-02-11T00:33:01.046 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-8 2021-02-11T00:33:01.096 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-9 2021-02-11T00:33:01.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:01.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:01.156 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:01.193 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:01.193 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:01.368 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:33:01.401 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003581.130865_83.1","last_update":"2021-02-11T00:33:01.130865Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:33:01.402 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:33:01.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:01.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:01.402 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:01.438 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:01.438 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:01.683 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003581.130865_83.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:01.130865Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:01.683 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:33:01.684 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:33:01.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:01.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:01.684 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:01.724 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:01.724 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:01.978 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003581.130865_83.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:01.130865Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:01.979 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:33:01.979 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:33:01.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:01.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:01.979 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:02.024 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:02.024 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:02.242 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003581.130865_83.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:01.130865Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:02.242 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:33:02.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:02.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:02.610 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:02.642 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:02.642 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:02.841 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:33:02.866 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003582.559316_89.1","last_update":"2021-02-11T00:33:02.559316Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:33:02.866 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:33:02.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:02.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:02.867 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:02.908 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:02.908 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:03.136 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:03.137 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:33:03.137 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:33:03.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:03.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:03.137 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:03.178 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:03.179 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:03.425 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:03.426 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:33:03.426 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:33:03.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:03.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:03.426 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:03.467 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:03.467 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:03.692 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:03.693 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:33:03.709 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2021-02-11T00:33:03.709 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-10 2021-02-11T00:33:03.744 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-11 2021-02-11T00:33:03.801 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-12 2021-02-11T00:33:03.930 INFO:tasks.rgw.c1.client.1.smithi057.stdout:2021-02-11T00:33:03.929+0000 7f2d717fa700 -1 data sync zone:e989c9f2 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T00:33:03.930 INFO:tasks.rgw.c1.client.1.smithi057.stdout:2021-02-11T00:33:03.929+0000 7f2d6fff7700 -1 data sync zone:e989c9f2 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T00:33:03.933 INFO:tasks.rgw.c2.client.0.smithi095.stdout:2021-02-11T00:33:03.932+0000 7f6a28ff9700 -1 data sync zone:e989c9f2 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T00:33:03.933 INFO:tasks.rgw.c2.client.0.smithi095.stdout:2021-02-11T00:33:03.932+0000 7f6a2a7fc700 -1 data sync zone:e989c9f2 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T00:33:03.934 INFO:tasks.rgw.c2.client.1.smithi095.stdout:2021-02-11T00:33:03.932+0000 7f22ff7fe700 -1 data sync zone:e989c9f2 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T00:33:03.935 INFO:tasks.rgw.c2.client.1.smithi095.stdout:2021-02-11T00:33:03.932+0000 7f23097fa700 -1 data sync zone:e989c9f2 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T00:33:04.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:04.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:04.447 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:04.491 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:04.491 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:04.662 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:33:04.685 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003582.559316_89.1","last_update":"2021-02-11T00:33:02.559316Z"},{"marker":"1_1613003583.836924_68.1","last_update":"2021-02-11T00:33:03.836924Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:33:04.685 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:33:04.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:04.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:04.686 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:04.727 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:04.727 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:04.953 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003583.836924_68.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:33:03.836924Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:04.953 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:33:04.954 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:33:04.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:04.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:04.954 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:04.995 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:04.995 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:05.256 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003583.836924_68.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:33:03.836924Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:08.291 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:33:08.291 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:33:08.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:08.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:08.292 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:08.332 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:08.332 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:08.545 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003583.836924_68.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:33:03.836924Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:08.546 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:33:08.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:08.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:08.546 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:08.633 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:08.634 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:08.852 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:08.848+0000 7f897ebf6080 1 bucket sync caught up with source: 2021-02-11T00:33:08.853 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000001.4.5, , 00000000001.4.5, , , , 00000000001.4.5, 00000000001.8.5, , ] 2021-02-11T00:33:08.853 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000001.4.5, , 00000000001.4.5, , , , 00000000001.4.5, 00000000001.8.5, , ] 2021-02-11T00:33:08.853 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:08.848+0000 7f897ebf6080 0 bucket checkpoint complete 2021-02-11T00:33:08.881 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-10 zones={test-zone2, test-zone1} 2021-02-11T00:33:08.893 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:08.900 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T00:33:08.900 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T00:33:08.900 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T00:33:08.901 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:08.901 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:08.905 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T00:33:08.905 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T00:33:08.905 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T00:33:08.905 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:08.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:08.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:09.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:09.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:09.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:09.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:09.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:09.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:09.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:09.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:09.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:09.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:10.013 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-10 zones={test-zone2, test-zone1} 2021-02-11T00:33:10.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:10.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:10.014 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:10.060 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:10.060 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:10.298 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:10.297+0000 7f7090d33080 1 bucket sync caught up with source: 2021-02-11T00:33:10.298 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000001.4.5, , 00000000001.4.5, , , , 00000000001.4.5, 00000000001.8.5, , ] 2021-02-11T00:33:10.298 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000001.4.5, , 00000000001.4.5, , , , 00000000001.4.5, 00000000001.8.5, , ] 2021-02-11T00:33:10.298 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:10.297+0000 7f7090d33080 0 bucket checkpoint complete 2021-02-11T00:33:10.326 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-10 zones={test-zone3, test-zone1} 2021-02-11T00:33:10.335 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:10.340 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T00:33:10.341 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T00:33:10.341 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T00:33:10.341 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:10.341 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:10.345 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T00:33:10.345 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T00:33:10.345 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T00:33:10.345 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:10.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:10.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:10.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:10.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:10.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:10.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:11.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:11.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:11.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:11.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:11.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:11.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:11.582 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-10 zones={test-zone3, test-zone1} 2021-02-11T00:33:11.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:11.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:11.582 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:11.631 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:11.631 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:11.833 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:11.832+0000 7f758138c080 1 bucket sync caught up with source: 2021-02-11T00:33:11.834 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000001.4.5, , 00000000001.4.5, , , , 00000000001.4.5, 00000000001.8.5, , ] 2021-02-11T00:33:11.834 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000001.4.5, , 00000000001.4.5, , , , 00000000001.4.5, 00000000001.8.5, , ] 2021-02-11T00:33:11.834 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:11.832+0000 7f758138c080 0 bucket checkpoint complete 2021-02-11T00:33:11.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:11.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:11.865 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:11.917 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:11.917 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:12.128 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:12.125+0000 7f2551068080 1 bucket sync caught up with source: 2021-02-11T00:33:12.128 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000001.9.5, , 00000000001.10.5, , , , 00000000001.9.5, 00000000001.8.5, , ] 2021-02-11T00:33:12.128 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000001.9.5, , 00000000001.10.5, , , , 00000000001.9.5, 00000000001.8.5, , ] 2021-02-11T00:33:12.128 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:12.125+0000 7f2551068080 0 bucket checkpoint complete 2021-02-11T00:33:12.162 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-11 zones={test-zone1, test-zone2} 2021-02-11T00:33:12.173 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:12.180 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T00:33:12.180 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T00:33:12.180 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T00:33:12.180 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:12.181 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:12.187 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T00:33:12.187 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T00:33:12.187 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T00:33:12.187 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:12.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:12.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:12.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:12.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:12.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:12.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:12.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:12.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:12.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:13.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:13.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:13.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:13.310 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-11 zones={test-zone1, test-zone2} 2021-02-11T00:33:13.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:13.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:13.310 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:13.358 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:13.358 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:13.605 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:13.605+0000 7fc975356080 1 bucket sync caught up with source: 2021-02-11T00:33:13.606 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000001.9.5, , 00000000001.10.5, , , , 00000000001.9.5, 00000000001.8.5, , ] 2021-02-11T00:33:13.606 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000001.9.5, , 00000000001.10.5, , , , 00000000001.9.5, 00000000001.8.5, , ] 2021-02-11T00:33:13.606 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:13.605+0000 7fc975356080 0 bucket checkpoint complete 2021-02-11T00:33:13.634 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-11 zones={test-zone3, test-zone2} 2021-02-11T00:33:13.644 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:13.650 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T00:33:13.650 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T00:33:13.650 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T00:33:13.650 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:13.651 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:13.656 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T00:33:13.656 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T00:33:13.656 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T00:33:13.656 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:13.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:13.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:13.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:13.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:13.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:14.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:14.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:14.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:14.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:14.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:14.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:14.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:14.766 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-11 zones={test-zone3, test-zone2} 2021-02-11T00:33:14.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:14.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:14.766 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:14.814 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:15.047 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:15.045+0000 7f5d57193080 1 bucket sync caught up with source: 2021-02-11T00:33:15.048 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.8.5, 00000000001.9.5, , ] 2021-02-11T00:33:15.048 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.8.5, 00000000001.9.5, , ] 2021-02-11T00:33:15.048 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:15.045+0000 7f5d57193080 0 bucket checkpoint complete 2021-02-11T00:33:15.076 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-12 zones={test-zone1, test-zone3} 2021-02-11T00:33:15.087 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:15.093 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T00:33:15.093 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T00:33:15.093 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T00:33:15.093 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:15.094 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:15.099 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T00:33:15.099 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T00:33:15.099 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T00:33:15.099 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:15.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:15.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:15.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:15.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:15.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:15.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:15.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:15.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:15.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:15.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:15.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:16.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:16.207 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-12 zones={test-zone1, test-zone3} 2021-02-11T00:33:16.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:16.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:16.208 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:16.256 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:16.256 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:16.473 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:16.472+0000 7f532580f080 1 bucket sync caught up with source: 2021-02-11T00:33:16.473 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.8.5, 00000000001.9.5, , ] 2021-02-11T00:33:16.473 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.8.5, 00000000001.9.5, , ] 2021-02-11T00:33:16.473 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:16.472+0000 7f532580f080 0 bucket checkpoint complete 2021-02-11T00:33:16.505 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-12 zones={test-zone2, test-zone3} 2021-02-11T00:33:16.515 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:16.522 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T00:33:16.522 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T00:33:16.522 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T00:33:16.522 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:16.522 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:16.529 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-02-11T00:33:16.529 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-02-11T00:33:16.529 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-02-11T00:33:16.529 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:16.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:16.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:16.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-02-11T00:33:16.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:16.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:16.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-02-11T00:33:17.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:17.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:17.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-02-11T00:33:17.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:17.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:17.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:17.641 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-12 zones={test-zone2, test-zone3} 2021-02-11T00:33:17.642 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2021-02-11T00:33:17.642 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-13 2021-02-11T00:33:17.671 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-14 2021-02-11T00:33:17.730 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-15 2021-02-11T00:33:17.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:17.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:17.935 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:17.971 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:17.972 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:18.154 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:33:18.176 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003582.559316_89.1","last_update":"2021-02-11T00:33:02.559316Z"},{"marker":"1_1613003597.771457_86.1","last_update":"2021-02-11T00:33:17.771457Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:33:18.176 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:33:18.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:18.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:18.177 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:18.221 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:18.221 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:18.430 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003597.771457_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:33:17.771457Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:18.430 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:33:18.431 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:33:18.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:18.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:18.431 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:18.473 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:18.473 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:18.689 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003597.771457_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:33:17.771457Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:18.690 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:33:18.690 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:33:18.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:18.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:18.691 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:18.733 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:18.734 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:18.953 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003597.771457_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:33:17.771457Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:18.954 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:33:18.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:18.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:18.955 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:19.048 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:19.049 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:19.268 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:19.268+0000 7f4e4afd8080 1 bucket sync caught up with source: 2021-02-11T00:33:19.268 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2021-02-11T00:33:19.268 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000001.11.5, , , , , , , , , ] 2021-02-11T00:33:19.268 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:19.268+0000 7f4e4afd8080 0 bucket checkpoint complete 2021-02-11T00:33:19.301 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-13 zones={test-zone2, test-zone1} 2021-02-11T00:33:19.311 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:19.359 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:19.359 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:19.366 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:19.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:19.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:19.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:19.690 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-13 zones={test-zone2, test-zone1} 2021-02-11T00:33:19.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:19.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:19.691 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:19.735 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:19.735 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:19.953 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:19.953+0000 7faad4a07080 1 bucket sync caught up with source: 2021-02-11T00:33:19.953 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2021-02-11T00:33:19.953 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000001.11.5, , , , , , , , , ] 2021-02-11T00:33:19.953 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:19.953+0000 7faad4a07080 0 bucket checkpoint complete 2021-02-11T00:33:19.978 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-13 zones={test-zone3, test-zone1} 2021-02-11T00:33:19.987 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:20.035 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:20.035 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:20.041 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:20.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:20.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:20.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:20.378 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-13 zones={test-zone3, test-zone1} 2021-02-11T00:33:20.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:20.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:20.378 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:20.427 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:20.427 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:20.651 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:20.648+0000 7f6efc5bd080 1 bucket sync caught up with source: 2021-02-11T00:33:20.651 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000001.11.5, , , , , , , , , ] 2021-02-11T00:33:20.651 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000001.11.5, , , , , , , , , ] 2021-02-11T00:33:20.651 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:20.648+0000 7f6efc5bd080 0 bucket checkpoint complete 2021-02-11T00:33:20.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:20.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:20.685 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:20.735 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:20.735 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:20.971 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:20.969+0000 7f8e512a6080 1 bucket sync caught up with source: 2021-02-11T00:33:20.971 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2021-02-11T00:33:20.971 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000001.9.5, , , , , , , , , ] 2021-02-11T00:33:20.971 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:20.969+0000 7f8e512a6080 0 bucket checkpoint complete 2021-02-11T00:33:21.008 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-14 zones={test-zone1, test-zone2} 2021-02-11T00:33:21.019 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:21.067 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:21.067 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:21.075 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:21.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:21.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:21.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:21.422 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-14 zones={test-zone1, test-zone2} 2021-02-11T00:33:21.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:21.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:21.423 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:21.471 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:21.472 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:21.711 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:21.709+0000 7f15bd3d5080 1 bucket sync caught up with source: 2021-02-11T00:33:21.711 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2021-02-11T00:33:21.711 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000001.9.5, , , , , , , , , ] 2021-02-11T00:33:21.711 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:21.709+0000 7f15bd3d5080 0 bucket checkpoint complete 2021-02-11T00:33:21.746 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-14 zones={test-zone3, test-zone2} 2021-02-11T00:33:21.757 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:21.807 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:21.807 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:21.814 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:21.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:21.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:22.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:22.138 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-14 zones={test-zone3, test-zone2} 2021-02-11T00:33:22.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:22.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:22.138 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:22.187 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:22.187 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:22.396 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:22.397+0000 7f4a21602080 1 bucket sync caught up with source: 2021-02-11T00:33:22.397 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000001.3.5, , , , , , , , , ] 2021-02-11T00:33:22.397 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000001.3.5, , , , , , , , , ] 2021-02-11T00:33:22.397 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:22.397+0000 7f4a21602080 0 bucket checkpoint complete 2021-02-11T00:33:22.423 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-15 zones={test-zone1, test-zone3} 2021-02-11T00:33:22.433 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:22.479 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:22.479 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:22.486 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:22.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:22.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:22.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:22.817 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-15 zones={test-zone1, test-zone3} 2021-02-11T00:33:22.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:22.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:22.817 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:22.865 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:22.865 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:23.076 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:23.076+0000 7f1870860080 1 bucket sync caught up with source: 2021-02-11T00:33:23.076 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000001.3.5, , , , , , , , , ] 2021-02-11T00:33:23.076 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000001.3.5, , , , , , , , , ] 2021-02-11T00:33:23.076 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:23.076+0000 7f1870860080 0 bucket checkpoint complete 2021-02-11T00:33:23.108 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-15 zones={test-zone2, test-zone3} 2021-02-11T00:33:23.118 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:23.167 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:23.167 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:23.174 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-02-11T00:33:23.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:23.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:23.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-02-11T00:33:23.535 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-15 zones={test-zone2, test-zone3} 2021-02-11T00:33:23.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:23.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:23.553 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:23.600 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:23.600 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:23.818 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:23.816+0000 7f4c25015080 1 bucket sync caught up with source: 2021-02-11T00:33:23.819 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000002.13.5, , , , , , , , , ] 2021-02-11T00:33:23.819 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2021-02-11T00:33:23.819 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:23.816+0000 7f4c25015080 0 bucket checkpoint complete 2021-02-11T00:33:23.852 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-13 zones={test-zone2, test-zone1} 2021-02-11T00:33:23.862 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:23.914 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:23.980 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-13 zones={test-zone2, test-zone1} 2021-02-11T00:33:23.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:23.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:23.981 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:24.026 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:24.026 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:24.223 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:24.221+0000 7fbba5193080 1 bucket sync caught up with source: 2021-02-11T00:33:24.223 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000002.13.5, , , , , , , , , ] 2021-02-11T00:33:24.223 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2021-02-11T00:33:24.223 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:24.221+0000 7fbba5193080 0 bucket checkpoint complete 2021-02-11T00:33:24.252 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-13 zones={test-zone3, test-zone1} 2021-02-11T00:33:24.261 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:24.307 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:24.372 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-13 zones={test-zone3, test-zone1} 2021-02-11T00:33:24.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:24.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:24.373 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:24.420 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:24.420 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:24.628 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:24.628+0000 7efe65aba080 1 bucket sync caught up with source: 2021-02-11T00:33:24.628 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000002.13.5, , , , , , , , , ] 2021-02-11T00:33:24.628 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2021-02-11T00:33:24.628 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:24.628+0000 7efe65aba080 0 bucket checkpoint complete 2021-02-11T00:33:24.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:24.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:24.677 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:24.731 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:24.731 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:24.935 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:24.933+0000 7f8761b0d080 1 bucket sync caught up with source: 2021-02-11T00:33:24.935 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000002.11.5, , , , , , , , , ] 2021-02-11T00:33:24.935 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000002.11.5, , , , , , , , , ] 2021-02-11T00:33:24.935 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:24.933+0000 7f8761b0d080 0 bucket checkpoint complete 2021-02-11T00:33:24.964 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-14 zones={test-zone1, test-zone2} 2021-02-11T00:33:24.974 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:25.023 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:25.084 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-14 zones={test-zone1, test-zone2} 2021-02-11T00:33:25.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:25.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:25.085 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:25.133 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:25.133 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:25.352 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:25.349+0000 7fb33a5ec080 1 bucket sync caught up with source: 2021-02-11T00:33:25.352 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000002.11.5, , , , , , , , , ] 2021-02-11T00:33:25.352 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000002.11.5, , , , , , , , , ] 2021-02-11T00:33:25.352 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:25.349+0000 7fb33a5ec080 0 bucket checkpoint complete 2021-02-11T00:33:25.387 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-14 zones={test-zone3, test-zone2} 2021-02-11T00:33:25.397 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:25.447 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:25.510 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-14 zones={test-zone3, test-zone2} 2021-02-11T00:33:25.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:25.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:25.527 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:25.571 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:25.571 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:25.804 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:25.805+0000 7f8e1a5fd080 1 bucket sync caught up with source: 2021-02-11T00:33:25.804 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000002.7.5, , , , , , , , , ] 2021-02-11T00:33:25.804 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000002.7.5, , , , , , , , , ] 2021-02-11T00:33:25.805 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:25.805+0000 7f8e1a5fd080 0 bucket checkpoint complete 2021-02-11T00:33:25.833 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-15 zones={test-zone1, test-zone3} 2021-02-11T00:33:25.842 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:25.891 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:25.952 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-15 zones={test-zone1, test-zone3} 2021-02-11T00:33:25.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:25.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:25.953 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:26.003 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:26.003 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:26.211 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:26.212+0000 7fa9749b5080 1 bucket sync caught up with source: 2021-02-11T00:33:26.211 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000002.7.5, , , , , , , , , ] 2021-02-11T00:33:26.211 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000002.7.5, , , , , , , , , ] 2021-02-11T00:33:26.212 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:26.212+0000 7fa9749b5080 0 bucket checkpoint complete 2021-02-11T00:33:26.243 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-15 zones={test-zone2, test-zone3} 2021-02-11T00:33:26.253 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:26.303 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:26.366 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-15 zones={test-zone2, test-zone3} 2021-02-11T00:33:26.366 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2021-02-11T00:33:26.367 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-16 2021-02-11T00:33:26.401 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-17 2021-02-11T00:33:26.457 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-18 2021-02-11T00:33:26.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:26.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:26.577 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:26.621 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:26.621 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:26.791 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:33:26.825 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003582.559316_89.1","last_update":"2021-02-11T00:33:02.559316Z"},{"marker":"1_1613003606.555218_111.1","last_update":"2021-02-11T00:33:26.555218Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:33:26.825 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:33:26.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:26.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:26.826 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:26.863 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:26.864 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:27.081 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003606.555218_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:33:26.555218Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:27.082 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:33:27.082 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:33:27.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:27.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:27.082 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:27.124 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:27.124 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:27.374 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003606.555218_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:33:26.555218Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:27.375 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:33:27.375 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:33:27.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:27.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:33:27.375 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:33:27.416 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:27.416 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:27.655 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003606.555218_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:33:26.555218Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:33:27.656 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:33:27.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:27.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:27.707 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:27.755 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:27.756 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:27.958 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:27.956+0000 7f97db7c4080 1 bucket sync caught up with source: 2021-02-11T00:33:27.959 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , 00000000002.13.10, , , , , ] 2021-02-11T00:33:27.959 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , 00000000002.13.10, , , , , ] 2021-02-11T00:33:27.959 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:27.956+0000 7f97db7c4080 0 bucket checkpoint complete 2021-02-11T00:33:27.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:27.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:27.989 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:28.037 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:28.037 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:28.216 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:28.217+0000 7fac06f95080 1 waiting for incremental sync to catch up: 2021-02-11T00:33:28.217 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , 00000000003.15.9, , , , , ] 2021-02-11T00:33:28.217 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , 00000000004.17.9, , , , , ] 2021-02-11T00:33:33.220 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:33.221+0000 7fac06f95080 1 bucket sync caught up with source: 2021-02-11T00:33:33.220 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , 00000000005.19.9, , , , , ] 2021-02-11T00:33:33.220 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , 00000000004.17.9, , , , , ] 2021-02-11T00:33:33.221 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:33.221+0000 7fac06f95080 0 bucket checkpoint complete 2021-02-11T00:33:33.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:33.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:33.257 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:33.303 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:33.303 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:33.526 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:33.524+0000 7f8e83d3b080 1 bucket sync caught up with source: 2021-02-11T00:33:33.526 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , 00000000005.19.9, , , , , ] 2021-02-11T00:33:33.526 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , 00000000005.19.9, , , , , ] 2021-02-11T00:33:33.526 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:33.524+0000 7f8e83d3b080 0 bucket checkpoint complete 2021-02-11T00:33:33.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:33.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:33.611 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:33.658 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:33.658 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:33.888 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:33.885+0000 7f0ee2b6a080 1 waiting for incremental sync to catch up: 2021-02-11T00:33:33.888 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , , , ] 2021-02-11T00:33:33.888 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , 00000000002.19.10, , , , , ] 2021-02-11T00:33:38.892 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:38.889+0000 7f0ee2b6a080 1 bucket sync caught up with source: 2021-02-11T00:33:38.892 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , 00000000005.25.9, , , , , ] 2021-02-11T00:33:38.893 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , 00000000002.19.10, , , , , ] 2021-02-11T00:33:38.893 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:38.889+0000 7f0ee2b6a080 0 bucket checkpoint complete 2021-02-11T00:33:38.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:38.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:38.918 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:38.962 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:38.962 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:39.197 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:39.197+0000 7f4bcc689080 1 bucket sync caught up with source: 2021-02-11T00:33:39.197 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , 00000000005.25.9, , , , , ] 2021-02-11T00:33:39.197 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , 00000000005.25.9, , , , , ] 2021-02-11T00:33:39.197 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:39.197+0000 7f4bcc689080 0 bucket checkpoint complete 2021-02-11T00:33:39.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:39.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:39.283 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:39.332 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:39.332 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:39.555 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:39.553+0000 7f87331c2080 1 bucket sync caught up with source: 2021-02-11T00:33:39.555 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , 00000000002.18.10, , , , , ] 2021-02-11T00:33:39.556 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , 00000000002.18.10, , , , , ] 2021-02-11T00:33:39.556 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:39.553+0000 7f87331c2080 0 bucket checkpoint complete 2021-02-11T00:33:39.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:39.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:39.581 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:39.630 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:39.630 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:39.827 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:39.824+0000 7f7d4216f080 1 waiting for incremental sync to catch up: 2021-02-11T00:33:39.827 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , 00000000005.24.9, , , , , ] 2021-02-11T00:33:39.827 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , 00000000006.26.9, , , , , ] 2021-02-11T00:33:44.832 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:44.832+0000 7f7d4216f080 1 bucket sync caught up with source: 2021-02-11T00:33:44.832 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , 00000000006.26.9, , , , , ] 2021-02-11T00:33:44.832 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , 00000000006.26.9, , , , , ] 2021-02-11T00:33:44.832 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:44.832+0000 7f7d4216f080 0 bucket checkpoint complete 2021-02-11T00:33:46.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:46.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:46.583 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:46.631 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:46.631 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:46.810 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:46.808+0000 7f44ee581080 1 bucket sync caught up with source: 2021-02-11T00:33:46.810 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000010.33.5, , 00000000003.15.9, , 00000000008.25.9, , , , , ] 2021-02-11T00:33:46.810 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000010.33.5, , 00000000003.15.9, , 00000000008.25.9, , , , , ] 2021-02-11T00:33:46.810 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:46.808+0000 7f44ee581080 0 bucket checkpoint complete 2021-02-11T00:33:46.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:46.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:46.840 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:46.888 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:46.888 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:47.113 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:47.113+0000 7fb567efa080 1 bucket sync caught up with source: 2021-02-11T00:33:47.113 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000010.33.5, , 00000000003.15.9, , 00000000008.25.9, , , , , ] 2021-02-11T00:33:47.113 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000010.33.5, , 00000000003.15.9, , 00000000008.25.9, , , , , ] 2021-02-11T00:33:47.113 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:47.113+0000 7fb567efa080 0 bucket checkpoint complete 2021-02-11T00:33:47.139 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:47.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:47.140 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:47.187 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:47.187 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:47.390 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:47.388+0000 7fa5da692080 1 bucket sync caught up with source: 2021-02-11T00:33:47.391 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000010.33.5, , 00000000003.15.9, , 00000000008.25.9, , , , , ] 2021-02-11T00:33:47.391 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000010.33.5, , 00000000003.15.9, , 00000000008.25.9, , , , , ] 2021-02-11T00:33:47.391 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:47.392+0000 7fa5da692080 0 bucket checkpoint complete 2021-02-11T00:33:47.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:47.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:47.424 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:47.472 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:47.472 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:47.714 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:47.713+0000 7fdd57812080 1 bucket sync caught up with source: 2021-02-11T00:33:47.714 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000003.28.9, , 00000000010.32.5, , 00000000007.29.9, , , , , ] 2021-02-11T00:33:47.714 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000003.28.9, , 00000000010.32.5, , 00000000007.29.9, , , , , ] 2021-02-11T00:33:47.714 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:47.713+0000 7fdd57812080 0 bucket checkpoint complete 2021-02-11T00:33:47.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:47.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:47.751 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:47.794 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:47.794 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:48.001 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:48.001+0000 7f0df393f080 1 bucket sync caught up with source: 2021-02-11T00:33:48.001 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000003.28.9, , 00000000010.32.5, , 00000000007.29.9, , , , , ] 2021-02-11T00:33:48.001 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000003.28.9, , 00000000010.32.5, , 00000000007.29.9, , , , , ] 2021-02-11T00:33:48.002 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:48.001+0000 7f0df393f080 0 bucket checkpoint complete 2021-02-11T00:33:48.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:48.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:48.036 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:48.125 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:48.125 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:48.360 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:48.361+0000 7fd5d7078080 1 bucket sync caught up with source: 2021-02-11T00:33:48.360 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000003.23.9, , 00000000003.16.9, , 00000000014.46.5, , , , , ] 2021-02-11T00:33:48.360 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000003.23.9, , 00000000003.16.9, , 00000000014.46.5, , , , , ] 2021-02-11T00:33:48.360 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:48.361+0000 7fd5d7078080 0 bucket checkpoint complete 2021-02-11T00:33:48.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:48.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:48.392 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:48.440 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:48.440 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:48.655 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:48.656+0000 7f77ee776080 1 bucket sync caught up with source: 2021-02-11T00:33:48.656 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000003.23.9, , 00000000003.16.9, , 00000000014.46.5, , , , , ] 2021-02-11T00:33:48.656 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000003.23.9, , 00000000003.16.9, , 00000000014.46.5, , , , , ] 2021-02-11T00:33:48.656 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:48.656+0000 7f77ee776080 0 bucket checkpoint complete 2021-02-11T00:33:48.689 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-16 zones={test-zone2, test-zone1} 2021-02-11T00:33:48.699 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:48.705 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:33:48.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:33:48.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:33:48.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:33:48.705 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:48.712 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:33:48.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:33:48.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:33:48.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:33:48.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:48.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:48.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:48.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:48.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:49.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:49.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:49.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:49.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:49.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:49.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:49.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:50.022 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-16 zones={test-zone2, test-zone1} 2021-02-11T00:33:50.022 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-16 zones={test-zone3, test-zone1} 2021-02-11T00:33:50.031 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:50.036 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:33:50.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:33:50.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:33:50.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:33:50.036 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:50.041 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:33:50.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:33:50.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:33:50.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:33:50.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:50.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:50.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:50.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:50.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:50.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:50.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:50.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:50.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:50.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:50.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:50.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:51.034 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-16 zones={test-zone3, test-zone1} 2021-02-11T00:33:51.034 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-16 zones={test-zone3, test-zone2} 2021-02-11T00:33:51.042 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:51.047 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:33:51.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:33:51.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:33:51.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:33:51.047 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:51.095 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:33:51.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:33:51.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:33:51.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:33:51.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:51.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:51.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:51.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:51.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:51.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:51.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:51.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:51.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:51.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:51.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:52.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:52.127 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-16 zones={test-zone3, test-zone2} 2021-02-11T00:33:52.128 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:52.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:52.128 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:52.171 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:52.171 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:52.394 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:52.392+0000 7ff7ecb93080 1 bucket sync caught up with source: 2021-02-11T00:33:52.394 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000010.26.5, , 00000000003.11.9, , 00000000007.21.9, , , , , ] 2021-02-11T00:33:52.394 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000010.26.5, , 00000000003.11.9, , 00000000007.21.9, , , , , ] 2021-02-11T00:33:52.394 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:52.392+0000 7ff7ecb93080 0 bucket checkpoint complete 2021-02-11T00:33:52.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:52.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:52.431 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:52.479 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:52.480 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:52.723 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:52.721+0000 7fb11fdcc080 1 bucket sync caught up with source: 2021-02-11T00:33:52.723 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000010.26.5, , 00000000003.11.9, , 00000000007.21.9, , , , , ] 2021-02-11T00:33:52.723 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000010.26.5, , 00000000003.11.9, , 00000000007.21.9, , , , , ] 2021-02-11T00:33:52.723 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:52.721+0000 7fb11fdcc080 0 bucket checkpoint complete 2021-02-11T00:33:52.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:52.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:52.753 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:52.800 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:52.800 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:53.001 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:53.000+0000 7f21f32c4080 1 bucket sync caught up with source: 2021-02-11T00:33:53.001 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000010.26.5, , 00000000003.11.9, , 00000000007.21.9, , , , , ] 2021-02-11T00:33:53.001 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000010.26.5, , 00000000003.11.9, , 00000000007.21.9, , , , , ] 2021-02-11T00:33:53.001 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:53.000+0000 7f21f32c4080 0 bucket checkpoint complete 2021-02-11T00:33:53.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:53.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:53.031 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:53.079 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:53.079 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:53.310 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:53.309+0000 7fc724bcf080 1 bucket sync caught up with source: 2021-02-11T00:33:53.311 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000003.17.9, , 00000000010.29.5, , 00000000008.31.9, , , , , ] 2021-02-11T00:33:53.311 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000003.17.9, , 00000000010.29.5, , 00000000008.31.9, , , , , ] 2021-02-11T00:33:53.311 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:53.309+0000 7fc724bcf080 0 bucket checkpoint complete 2021-02-11T00:33:53.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:53.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:53.344 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:53.390 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:53.390 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:53.622 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:53.621+0000 7f00439dd080 1 bucket sync caught up with source: 2021-02-11T00:33:53.623 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000003.17.9, , 00000000010.29.5, , 00000000008.31.9, , , , , ] 2021-02-11T00:33:53.623 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000003.17.9, , 00000000010.29.5, , 00000000008.31.9, , , , , ] 2021-02-11T00:33:53.623 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:53.621+0000 7f00439dd080 0 bucket checkpoint complete 2021-02-11T00:33:53.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:53.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:53.663 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:53.751 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:53.752 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:53.985 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:53.985+0000 7fbbd6b48080 1 bucket sync caught up with source: 2021-02-11T00:33:53.986 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000003.18.9, , 00000000003.15.9, , 00000000014.37.5, , , , , ] 2021-02-11T00:33:53.986 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000003.18.9, , 00000000003.15.9, , 00000000014.37.5, , , , , ] 2021-02-11T00:33:53.986 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:53.985+0000 7fbbd6b48080 0 bucket checkpoint complete 2021-02-11T00:33:54.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:54.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:54.016 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:54.063 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:54.063 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:54.284 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:54.284+0000 7f3a5e833080 1 bucket sync caught up with source: 2021-02-11T00:33:54.284 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000003.18.9, , 00000000003.15.9, , 00000000014.37.5, , , , , ] 2021-02-11T00:33:54.284 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000003.18.9, , 00000000003.15.9, , 00000000014.37.5, , , , , ] 2021-02-11T00:33:54.284 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:54.284+0000 7f3a5e833080 0 bucket checkpoint complete 2021-02-11T00:33:54.316 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-17 zones={test-zone2, test-zone1} 2021-02-11T00:33:54.328 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:54.335 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:33:54.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:33:54.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:33:54.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:33:54.335 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:54.342 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:33:54.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:33:54.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:33:54.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:33:54.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:54.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:54.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:54.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:54.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:54.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:54.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:54.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:55.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:55.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:55.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:55.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:55.511 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-17 zones={test-zone2, test-zone1} 2021-02-11T00:33:55.511 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-17 zones={test-zone3, test-zone1} 2021-02-11T00:33:55.522 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:55.528 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:33:55.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:33:55.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:33:55.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:33:55.529 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:55.534 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:33:55.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:33:55.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:33:55.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:33:55.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:55.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:55.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:55.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:55.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:55.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:55.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:56.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:56.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:56.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:56.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:56.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:56.510 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-17 zones={test-zone3, test-zone1} 2021-02-11T00:33:56.510 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-17 zones={test-zone3, test-zone2} 2021-02-11T00:33:56.521 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:56.527 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:33:56.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:33:56.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:33:56.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:33:56.527 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:56.575 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:33:56.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:33:56.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:33:56.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:33:56.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:56.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:56.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:56.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:56.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:56.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:57.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:57.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:57.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:33:57.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:57.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:57.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:33:57.621 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-17 zones={test-zone3, test-zone2} 2021-02-11T00:33:57.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:57.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:57.622 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:57.669 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:57.669 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:57.903 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:57.904+0000 7f5c7f5ca080 1 bucket sync caught up with source: 2021-02-11T00:33:57.903 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000010.45.5, , 00000000003.17.9, , 00000000007.21.9, , , , , ] 2021-02-11T00:33:57.903 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000010.45.5, , 00000000003.17.9, , 00000000007.21.9, , , , , ] 2021-02-11T00:33:57.903 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:57.904+0000 7f5c7f5ca080 0 bucket checkpoint complete 2021-02-11T00:33:57.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:57.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:57.934 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:57.983 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:57.983 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:58.216 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:58.217+0000 7f1f744a5080 1 bucket sync caught up with source: 2021-02-11T00:33:58.216 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000010.45.5, , 00000000003.17.9, , 00000000007.21.9, , , , , ] 2021-02-11T00:33:58.216 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000010.45.5, , 00000000003.17.9, , 00000000007.21.9, , , , , ] 2021-02-11T00:33:58.216 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:58.217+0000 7f1f744a5080 0 bucket checkpoint complete 2021-02-11T00:33:58.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:58.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:58.245 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:58.291 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:58.291 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:58.499 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:58.500+0000 7fd010224080 1 bucket sync caught up with source: 2021-02-11T00:33:58.499 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000010.45.5, , 00000000003.17.9, , 00000000007.21.9, , , , , ] 2021-02-11T00:33:58.500 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000010.45.5, , 00000000003.17.9, , 00000000007.21.9, , , , , ] 2021-02-11T00:33:58.500 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:58.500+0000 7fd010224080 0 bucket checkpoint complete 2021-02-11T00:33:58.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:58.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:58.532 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:58.580 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:58.581 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:58.805 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:58.805+0000 7f7772534080 1 bucket sync caught up with source: 2021-02-11T00:33:58.805 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000003.35.9, , 00000000010.38.5, , 00000000007.24.9, , , , , ] 2021-02-11T00:33:58.805 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000003.35.9, , 00000000010.38.5, , 00000000007.24.9, , , , , ] 2021-02-11T00:33:58.805 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:58.805+0000 7f7772534080 0 bucket checkpoint complete 2021-02-11T00:33:58.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:58.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:58.841 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:58.884 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:58.884 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:59.102 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:59.101+0000 7f4273052080 1 bucket sync caught up with source: 2021-02-11T00:33:59.102 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000003.35.9, , 00000000010.38.5, , 00000000007.24.9, , , , , ] 2021-02-11T00:33:59.102 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000003.35.9, , 00000000010.38.5, , 00000000007.24.9, , , , , ] 2021-02-11T00:33:59.102 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:59.101+0000 7f4273052080 0 bucket checkpoint complete 2021-02-11T00:33:59.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:59.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:59.136 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:59.228 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:59.228 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:59.446 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:59.445+0000 7f0261a65080 1 bucket sync caught up with source: 2021-02-11T00:33:59.446 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000003.52.9, , 00000000003.28.9, , 00000000016.46.5, , , , , ] 2021-02-11T00:33:59.446 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000003.52.9, , 00000000003.28.9, , 00000000016.46.5, , , , , ] 2021-02-11T00:33:59.446 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:33:59.445+0000 7f0261a65080 0 bucket checkpoint complete 2021-02-11T00:33:59.476 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:59.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:33:59.477 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:33:59.525 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:33:59.525 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:33:59.736 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:59.736+0000 7ffb1b33e080 1 bucket sync caught up with source: 2021-02-11T00:33:59.737 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000003.52.9, , 00000000003.28.9, , 00000000016.46.5, , , , , ] 2021-02-11T00:33:59.737 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000003.52.9, , 00000000003.28.9, , 00000000016.46.5, , , , , ] 2021-02-11T00:33:59.737 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:33:59.736+0000 7ffb1b33e080 0 bucket checkpoint complete 2021-02-11T00:33:59.770 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-18 zones={test-zone2, test-zone1} 2021-02-11T00:33:59.780 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:33:59.785 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:33:59.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:33:59.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:33:59.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:33:59.786 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:33:59.791 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:33:59.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:33:59.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:33:59.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:33:59.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:59.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:59.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:33:59.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:33:59.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:00.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:00.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:00.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:00.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:00.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:00.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:00.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:00.786 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-18 zones={test-zone2, test-zone1} 2021-02-11T00:34:00.787 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-18 zones={test-zone3, test-zone1} 2021-02-11T00:34:00.796 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:34:00.800 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:00.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:00.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:00.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:00.801 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:34:00.805 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:00.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:00.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:00.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:00.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:00.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:00.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:00.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:01.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:01.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:01.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:01.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:01.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:01.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:01.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:01.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:01.806 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-18 zones={test-zone3, test-zone1} 2021-02-11T00:34:01.806 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-18 zones={test-zone3, test-zone2} 2021-02-11T00:34:01.816 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:34:01.821 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:01.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:01.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:01.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:01.822 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:34:01.827 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:01.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:01.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:01.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:01.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:01.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:01.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:01.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:02.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:02.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:02.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:02.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:02.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:02.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:02.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:02.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:02.810 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-18 zones={test-zone3, test-zone2} 2021-02-11T00:34:03.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:03.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:03.429 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:03.476 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:03.476 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:03.676 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:03.676+0000 7faf27ea9080 1 bucket sync caught up with source: 2021-02-11T00:34:03.676 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000011.40.5, , 00000000004.18.5, , 00000000009.52.5, , , , , ] 2021-02-11T00:34:03.676 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000011.40.5, , 00000000004.18.5, , 00000000009.52.5, , , , , ] 2021-02-11T00:34:03.676 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:03.676+0000 7faf27ea9080 0 bucket checkpoint complete 2021-02-11T00:34:03.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:03.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:03.709 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:03.755 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:03.755 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:03.942 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:03.941+0000 7f908f2c9080 1 bucket sync caught up with source: 2021-02-11T00:34:03.942 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000011.40.5, , 00000000004.18.5, , 00000000009.52.5, , , , , ] 2021-02-11T00:34:03.942 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000011.40.5, , 00000000004.18.5, , 00000000009.52.5, , , , , ] 2021-02-11T00:34:03.942 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:03.941+0000 7f908f2c9080 0 bucket checkpoint complete 2021-02-11T00:34:03.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:03.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:03.964 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:04.011 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:04.011 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:04.216 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:04.216+0000 7f6eeb574080 1 bucket sync caught up with source: 2021-02-11T00:34:04.216 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000011.40.5, , 00000000004.18.5, , 00000000009.52.5, , , , , ] 2021-02-11T00:34:04.216 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000011.40.5, , 00000000004.18.5, , 00000000009.52.5, , , , , ] 2021-02-11T00:34:04.217 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:04.216+0000 7f6eeb574080 0 bucket checkpoint complete 2021-02-11T00:34:04.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:04.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:04.250 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:04.300 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:04.300 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:04.523 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:04.521+0000 7fc6c3bfb080 1 bucket sync caught up with source: 2021-02-11T00:34:04.523 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000004.31.5, , 00000000011.38.5, , 00000000008.38.5, , , , , ] 2021-02-11T00:34:04.523 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000004.31.5, , 00000000011.38.5, , 00000000008.38.5, , , , , ] 2021-02-11T00:34:04.524 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:04.521+0000 7fc6c3bfb080 0 bucket checkpoint complete 2021-02-11T00:34:04.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:04.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:04.558 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:04.604 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:04.604 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:04.824 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:04.825+0000 7fed08a1d080 1 bucket sync caught up with source: 2021-02-11T00:34:06.802 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000004.31.5, , 00000000011.38.5, , 00000000008.38.5, , , , , ] 2021-02-11T00:34:06.802 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000004.31.5, , 00000000011.38.5, , 00000000008.38.5, , , , , ] 2021-02-11T00:34:06.802 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:04.825+0000 7fed08a1d080 0 bucket checkpoint complete 2021-02-11T00:34:06.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:06.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:06.804 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:06.855 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:06.855 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:07.073 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:07.073+0000 7fabd6947080 1 bucket sync caught up with source: 2021-02-11T00:34:07.073 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000004.26.5, , 00000000004.19.5, , 00000000015.55.5, , , , , ] 2021-02-11T00:34:07.074 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000004.26.5, , 00000000004.19.5, , 00000000015.55.5, , , , , ] 2021-02-11T00:34:07.074 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:07.073+0000 7fabd6947080 0 bucket checkpoint complete 2021-02-11T00:34:07.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:07.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:07.105 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:07.152 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:07.153 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:07.372 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:07.372+0000 7f644ef9f080 1 bucket sync caught up with source: 2021-02-11T00:34:07.372 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000004.26.5, , 00000000004.19.5, , 00000000015.55.5, , , , , ] 2021-02-11T00:34:07.373 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000004.26.5, , 00000000004.19.5, , 00000000015.55.5, , , , , ] 2021-02-11T00:34:07.373 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:07.372+0000 7f644ef9f080 0 bucket checkpoint complete 2021-02-11T00:34:07.411 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-16 zones={test-zone2, test-zone1} 2021-02-11T00:34:07.422 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:34:07.428 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:07.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:07.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:07.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:07.428 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:34:07.435 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:07.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:07.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:07.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:07.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:07.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:07.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:07.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:07.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:07.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:07.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:07.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:08.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:08.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:08.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:08.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:08.421 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-16 zones={test-zone2, test-zone1} 2021-02-11T00:34:08.421 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-16 zones={test-zone3, test-zone1} 2021-02-11T00:34:08.431 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:34:08.436 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:08.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:08.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:08.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:08.436 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:34:08.440 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:08.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:08.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:08.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:08.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:08.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:08.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:08.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:08.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:08.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:08.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:08.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:09.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:09.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:09.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:09.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:09.405 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-16 zones={test-zone3, test-zone1} 2021-02-11T00:34:09.406 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-16 zones={test-zone3, test-zone2} 2021-02-11T00:34:09.419 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:34:09.423 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:09.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:09.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:09.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:09.424 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:34:09.430 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:09.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:09.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:09.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:09.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:09.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:09.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:09.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:09.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:09.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:09.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:09.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:10.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:10.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:10.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:10.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:10.457 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-16 zones={test-zone3, test-zone2} 2021-02-11T00:34:10.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:10.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:10.458 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:10.501 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:10.501 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:10.734 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:10.732+0000 7fb24a3fc080 1 bucket sync caught up with source: 2021-02-11T00:34:10.734 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000011.33.5, , 00000000004.14.5, , 00000000008.24.5, , , , , ] 2021-02-11T00:34:10.734 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000011.33.5, , 00000000004.14.5, , 00000000008.24.5, , , , , ] 2021-02-11T00:34:10.734 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:10.732+0000 7fb24a3fc080 0 bucket checkpoint complete 2021-02-11T00:34:10.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:10.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:10.767 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:10.815 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:10.815 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:11.029 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:11.029+0000 7f3873b2b080 1 bucket sync caught up with source: 2021-02-11T00:34:11.029 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000011.33.5, , 00000000004.14.5, , 00000000008.24.5, , , , , ] 2021-02-11T00:34:11.029 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000011.33.5, , 00000000004.14.5, , 00000000008.24.5, , , , , ] 2021-02-11T00:34:11.029 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:11.029+0000 7f3873b2b080 0 bucket checkpoint complete 2021-02-11T00:34:11.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:11.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:11.059 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:11.108 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:11.108 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:11.314 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:11.312+0000 7f4086a19080 1 bucket sync caught up with source: 2021-02-11T00:34:11.314 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000011.33.5, , 00000000004.14.5, , 00000000008.24.5, , , , , ] 2021-02-11T00:34:11.314 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000011.33.5, , 00000000004.14.5, , 00000000008.24.5, , , , , ] 2021-02-11T00:34:11.314 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:11.312+0000 7f4086a19080 0 bucket checkpoint complete 2021-02-11T00:34:11.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:11.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:11.346 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:11.392 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:11.392 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:11.623 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:11.621+0000 7f818c52e080 1 bucket sync caught up with source: 2021-02-11T00:34:11.623 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000004.20.5, , 00000000011.35.5, , 00000000009.34.5, , , , , ] 2021-02-11T00:34:11.623 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000004.20.5, , 00000000011.35.5, , 00000000009.34.5, , , , , ] 2021-02-11T00:34:11.623 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:11.621+0000 7f818c52e080 0 bucket checkpoint complete 2021-02-11T00:34:11.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:11.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:11.659 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:11.703 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:11.703 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:11.923 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:11.921+0000 7f8c295ca080 1 bucket sync caught up with source: 2021-02-11T00:34:11.923 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000004.20.5, , 00000000011.35.5, , 00000000009.34.5, , , , , ] 2021-02-11T00:34:11.923 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000004.20.5, , 00000000011.35.5, , 00000000009.34.5, , , , , ] 2021-02-11T00:34:11.923 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:11.921+0000 7f8c295ca080 0 bucket checkpoint complete 2021-02-11T00:34:11.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:11.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:11.956 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:12.047 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:12.047 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:12.279 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:12.277+0000 7fb1e264e080 1 bucket sync caught up with source: 2021-02-11T00:34:12.279 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000004.21.5, , 00000000004.18.5, , 00000000015.40.5, , , , , ] 2021-02-11T00:34:12.279 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000004.21.5, , 00000000004.18.5, , 00000000015.40.5, , , , , ] 2021-02-11T00:34:12.279 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:12.277+0000 7fb1e264e080 0 bucket checkpoint complete 2021-02-11T00:34:12.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:12.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:12.308 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:12.355 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:12.355 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:12.583 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:12.584+0000 7fd24bba0080 1 bucket sync caught up with source: 2021-02-11T00:34:12.583 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000004.21.5, , 00000000004.18.5, , 00000000015.40.5, , , , , ] 2021-02-11T00:34:12.583 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000004.21.5, , 00000000004.18.5, , 00000000015.40.5, , , , , ] 2021-02-11T00:34:12.583 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:12.584+0000 7fd24bba0080 0 bucket checkpoint complete 2021-02-11T00:34:12.616 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-17 zones={test-zone2, test-zone1} 2021-02-11T00:34:12.628 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:34:12.634 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:12.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:12.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:12.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:12.635 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:34:12.642 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:12.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:12.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:12.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:12.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:12.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:12.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:12.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:12.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:12.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:13.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:13.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:13.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:13.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:13.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:13.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:13.631 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-17 zones={test-zone2, test-zone1} 2021-02-11T00:34:13.631 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-17 zones={test-zone3, test-zone1} 2021-02-11T00:34:13.640 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:34:13.645 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:13.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:13.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:13.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:13.645 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:34:13.650 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:13.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:13.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:13.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:13.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:13.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:13.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:13.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:13.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:13.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:14.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:14.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:14.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:14.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:14.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:14.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:14.611 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-17 zones={test-zone3, test-zone1} 2021-02-11T00:34:14.612 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-17 zones={test-zone3, test-zone2} 2021-02-11T00:34:14.623 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:34:14.629 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:14.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:14.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:14.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:14.629 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:34:14.637 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:14.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:14.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:14.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:14.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:14.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:14.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:14.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:14.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:14.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:15.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:15.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:15.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:15.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:15.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:15.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:15.642 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-17 zones={test-zone3, test-zone2} 2021-02-11T00:34:15.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:15.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:15.643 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:15.687 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:15.687 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:15.898 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:15.900+0000 7fa82f4dc080 1 bucket sync caught up with source: 2021-02-11T00:34:15.899 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000011.54.5, , 00000000004.20.5, , 00000000008.24.5, , , , , ] 2021-02-11T00:34:15.899 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000011.54.5, , 00000000004.20.5, , 00000000008.24.5, , , , , ] 2021-02-11T00:34:15.899 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:15.900+0000 7fa82f4dc080 0 bucket checkpoint complete 2021-02-11T00:34:15.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:15.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:15.937 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:15.987 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:15.987 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:16.205+0000 7fcc87162080 1 bucket sync caught up with source: 2021-02-11T00:34:16.206 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000011.54.5, , 00000000004.20.5, , 00000000008.24.5, , , , , ] 2021-02-11T00:34:16.206 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000011.54.5, , 00000000004.20.5, , 00000000008.24.5, , , , , ] 2021-02-11T00:34:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:16.205+0000 7fcc87162080 0 bucket checkpoint complete 2021-02-11T00:34:16.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:16.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:16.236 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:16.283 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:16.284 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:16.495 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:16.496+0000 7f4ba21a2080 1 bucket sync caught up with source: 2021-02-11T00:34:16.495 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000011.54.5, , 00000000004.20.5, , 00000000008.24.5, , , , , ] 2021-02-11T00:34:16.495 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000011.54.5, , 00000000004.20.5, , 00000000008.24.5, , , , , ] 2021-02-11T00:34:16.495 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:16.496+0000 7f4ba21a2080 0 bucket checkpoint complete 2021-02-11T00:34:16.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:16.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:16.528 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:16.577 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:16.577 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:16.817 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:16.817+0000 7ffa31157080 1 bucket sync caught up with source: 2021-02-11T00:34:16.817 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000004.40.5, , 00000000011.45.5, , 00000000008.27.5, , , , , ] 2021-02-11T00:34:16.817 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000004.40.5, , 00000000011.45.5, , 00000000008.27.5, , , , , ] 2021-02-11T00:34:16.817 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:16.817+0000 7ffa31157080 0 bucket checkpoint complete 2021-02-11T00:34:16.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:16.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:16.851 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:16.896 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:16.896 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:17.127 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:17.125+0000 7f720886a080 1 bucket sync caught up with source: 2021-02-11T00:34:17.127 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000004.40.5, , 00000000011.45.5, , 00000000008.27.5, , , , , ] 2021-02-11T00:34:17.127 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000004.40.5, , 00000000011.45.5, , 00000000008.27.5, , , , , ] 2021-02-11T00:34:17.127 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:17.125+0000 7f720886a080 0 bucket checkpoint complete 2021-02-11T00:34:17.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:17.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:17.162 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:17.206 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:17.207 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:17.432 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:17.433+0000 7fb32d221080 1 bucket sync caught up with source: 2021-02-11T00:34:17.432 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, 00000000004.57.5, , 00000000004.31.5, , 00000000017.49.5, , , , , ] 2021-02-11T00:34:17.432 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, 00000000004.57.5, , 00000000004.31.5, , 00000000017.49.5, , , , , ] 2021-02-11T00:34:17.432 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:17.433+0000 7fb32d221080 0 bucket checkpoint complete 2021-02-11T00:34:17.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:17.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:17.464 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:17.512 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:17.513 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:17.739 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:17.740+0000 7fb875073080 1 bucket sync caught up with source: 2021-02-11T00:34:17.739 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, 00000000004.57.5, , 00000000004.31.5, , 00000000017.49.5, , , , , ] 2021-02-11T00:34:17.739 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, 00000000004.57.5, , 00000000004.31.5, , 00000000017.49.5, , , , , ] 2021-02-11T00:34:17.739 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:17.740+0000 7fb875073080 0 bucket checkpoint complete 2021-02-11T00:34:17.771 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-18 zones={test-zone2, test-zone1} 2021-02-11T00:34:17.783 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:34:17.835 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:17.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:17.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:17.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:17.835 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:34:17.843 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:17.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:17.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:17.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:17.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:17.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:17.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:18.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:18.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:18.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:18.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:18.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:18.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:19.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:19.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:19.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:19.337 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-18 zones={test-zone2, test-zone1} 2021-02-11T00:34:19.338 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-18 zones={test-zone3, test-zone1} 2021-02-11T00:34:19.347 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:34:19.352 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:19.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:19.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:19.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:19.353 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:34:19.357 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:19.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:19.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:19.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:19.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:19.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:19.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:19.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:19.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:19.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:19.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:19.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:19.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:21.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:21.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:21.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:21.849 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-18 zones={test-zone3, test-zone1} 2021-02-11T00:34:21.849 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-18 zones={test-zone3, test-zone2} 2021-02-11T00:34:21.873 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:34:21.887 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:21.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:21.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:21.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:21.887 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:34:21.909 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:21.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-02-11T00:34:21.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-02-11T00:34:21.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-02-11T00:34:21.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:21.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:22.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:22.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:22.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:22.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-02-11T00:34:22.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:22.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:22.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-02-11T00:34:22.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:22.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:22.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-02-11T00:34:23.009 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-18 zones={test-zone3, test-zone2} 2021-02-11T00:34:23.010 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2021-02-11T00:34:23.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:34:23.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:34:23.064 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:34:23.102 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:23.102 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:23.307 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:34:23.348 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003582.559316_89.1","last_update":"2021-02-11T00:33:02.559316Z"},{"marker":"1_1613003663.066081_303.1","last_update":"2021-02-11T00:34:23.066081Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:34:23.348 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:34:23.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:34:23.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:34:23.349 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:34:23.390 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:23.390 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:23.616 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003663.066081_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.066081Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:34:23.617 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:34:23.617 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:34:23.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:34:23.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:34:23.618 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:34:23.662 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:23.662 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:23.886 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003663.066081_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.066081Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:34:23.886 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:34:23.886 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:34:23.886 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:34:23.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:34:23.887 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:34:23.928 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:23.928 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:24.136 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003663.066081_303.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.066081Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:34:24.137 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:34:24.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:24.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:24.187 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:24.234 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:24.235 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:24.441 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:24.439+0000 7f6ffaac7080 1 bucket sync caught up with source: 2021-02-11T00:34:24.441 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , 00000000002.9.10, , , , , ] 2021-02-11T00:34:24.441 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , 00000000002.9.10, , , , , ] 2021-02-11T00:34:24.441 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:24.439+0000 7f6ffaac7080 0 bucket checkpoint complete 2021-02-11T00:34:24.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:24.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:24.479 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:24.521 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:24.521 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:24.702 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:24.704+0000 7f9842649080 1 bucket sync caught up with source: 2021-02-11T00:34:24.702 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , 00000000005.15.9, , , , , ] 2021-02-11T00:34:24.702 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , 00000000005.15.9, , , , , ] 2021-02-11T00:34:24.702 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:24.704+0000 7f9842649080 0 bucket checkpoint complete 2021-02-11T00:34:24.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:24.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:24.734 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:24.785 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:24.786 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:24.981 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:24.979+0000 7f82b1990080 1 bucket sync caught up with source: 2021-02-11T00:34:24.981 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , 00000000006.17.9, , , , , ] 2021-02-11T00:34:24.981 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , 00000000006.17.9, , , , , ] 2021-02-11T00:34:24.981 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:24.979+0000 7f82b1990080 0 bucket checkpoint complete 2021-02-11T00:34:25.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:25.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:25.016 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:25.063 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:25.064 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:25.298 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:25.300+0000 7f9e0bf14080 1 bucket sync caught up with source: 2021-02-11T00:34:25.298 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , 00000000004.19.9, , , , , ] 2021-02-11T00:34:25.298 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , 00000000004.19.9, , , , , ] 2021-02-11T00:34:25.298 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:25.300+0000 7f9e0bf14080 0 bucket checkpoint complete 2021-02-11T00:34:25.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:25.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:25.334 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:25.382 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:25.383 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:25.600 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:25.600+0000 7efca0afc080 1 bucket sync caught up with source: 2021-02-11T00:34:25.600 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , 00000000004.19.9, , , , , ] 2021-02-11T00:34:25.600 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , 00000000004.19.9, , , , , ] 2021-02-11T00:34:25.600 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:25.600+0000 7efca0afc080 0 bucket checkpoint complete 2021-02-11T00:34:25.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:25.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:25.637 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:25.685 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:25.685 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:25.907 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:25.908+0000 7f1eddccb080 1 bucket sync caught up with source: 2021-02-11T00:34:25.907 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , 00000000004.20.9, , , , , ] 2021-02-11T00:34:25.907 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , 00000000004.20.9, , , , , ] 2021-02-11T00:34:25.908 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:25.908+0000 7f1eddccb080 0 bucket checkpoint complete 2021-02-11T00:34:25.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:25.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:25.936 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:25.984 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:25.984 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:26.207 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:26.211+0000 7f9df7877080 1 bucket sync caught up with source: 2021-02-11T00:34:26.207 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , 00000000004.20.9, , , , , ] 2021-02-11T00:34:26.207 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , 00000000004.20.9, , , , , ] 2021-02-11T00:34:26.207 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:26.211+0000 7f9df7877080 0 bucket checkpoint complete 2021-02-11T00:34:26.239 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-19 zones={test-zone2, test-zone1} 2021-02-11T00:34:26.249 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:34:26.299 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:26.299 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:34:26.306 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:26.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:26.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:26.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:26.501 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-19 zones={test-zone2, test-zone1} 2021-02-11T00:34:26.501 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-19 zones={test-zone3, test-zone1} 2021-02-11T00:34:26.511 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:34:26.559 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:26.559 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:34:26.566 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:26.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:26.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:26.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:26.763 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-19 zones={test-zone3, test-zone1} 2021-02-11T00:34:26.763 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-19 zones={test-zone3, test-zone2} 2021-02-11T00:34:26.772 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:34:26.819 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:26.819 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:34:26.825 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:26.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:26.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:26.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:27.018 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-19 zones={test-zone3, test-zone2} 2021-02-11T00:34:28.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:28.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:28.523 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:28.579 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:28.579 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:28.788 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:28.791+0000 7ff410918080 1 bucket sync caught up with source: 2021-02-11T00:34:28.788 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000051.133.8, , ] 2021-02-11T00:34:28.788 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000051.133.8, , ] 2021-02-11T00:34:28.788 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:28.791+0000 7ff410918080 0 bucket checkpoint complete 2021-02-11T00:34:28.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:28.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:28.813 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:28.858 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:28.859 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:29.039 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:29.040+0000 7f9e2786d080 1 waiting for incremental sync to catch up: 2021-02-11T00:34:29.039 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000051.133.8, , ] 2021-02-11T00:34:29.039 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000056.144.9, , ] 2021-02-11T00:34:34.044 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:34.043+0000 7f9e2786d080 1 bucket sync caught up with source: 2021-02-11T00:34:34.044 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000056.144.9, , ] 2021-02-11T00:34:34.044 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000056.144.9, , ] 2021-02-11T00:34:34.045 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:34.043+0000 7f9e2786d080 0 bucket checkpoint complete 2021-02-11T00:34:34.086 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:34.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:34.086 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:34.131 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:34.132 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:34.346 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:34.346+0000 7f53ed87f080 1 bucket sync caught up with source: 2021-02-11T00:34:34.346 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000056.144.9, , ] 2021-02-11T00:34:34.347 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000056.144.9, , ] 2021-02-11T00:34:34.347 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:34.346+0000 7f53ed87f080 0 bucket checkpoint complete 2021-02-11T00:34:34.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:34.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:34.378 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:34.426 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:34.426 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:34.651+0000 7f4ba560b080 1 bucket sync caught up with source: 2021-02-11T00:34:34.650 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000052.138.9, , ] 2021-02-11T00:34:34.650 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000052.138.9, , ] 2021-02-11T00:34:34.651 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:34.651+0000 7f4ba560b080 0 bucket checkpoint complete 2021-02-11T00:34:34.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:34.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:34.687 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:34.734 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:34.735 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:34.945 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:34.943+0000 7fc22d830080 1 bucket sync caught up with source: 2021-02-11T00:34:34.945 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000052.138.9, , ] 2021-02-11T00:34:34.945 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000052.138.9, , ] 2021-02-11T00:34:34.945 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:34.943+0000 7fc22d830080 0 bucket checkpoint complete 2021-02-11T00:34:34.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:34.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:34.979 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:35.024 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:35.024 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:35.258 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:35.259+0000 7f0d9bdf9080 1 bucket sync caught up with source: 2021-02-11T00:34:35.258 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , 00000000004.20.9, , , 00000000055.145.9, , ] 2021-02-11T00:34:35.258 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , 00000000004.20.9, , , 00000000055.145.9, , ] 2021-02-11T00:34:35.258 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:34:35.259+0000 7f0d9bdf9080 0 bucket checkpoint complete 2021-02-11T00:34:35.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:35.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:34:35.287 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:34:35.335 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:34:35.335 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:34:35.572 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:35.574+0000 7f6850fa7080 1 bucket sync caught up with source: 2021-02-11T00:34:35.573 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , 00000000004.20.9, , , 00000000055.145.9, , ] 2021-02-11T00:34:35.573 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , 00000000004.20.9, , , 00000000055.145.9, , ] 2021-02-11T00:34:35.573 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:34:35.574+0000 7f6850fa7080 0 bucket checkpoint complete 2021-02-11T00:34:35.611 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-19 zones={test-zone2, test-zone1} 2021-02-11T00:34:35.621 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:34:35.629 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:35.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.632 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:34:35.641 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:35.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:35.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:35.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:35.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:35.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:35.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:35.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:36.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:36.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:36.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:36.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:36.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:36.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:36.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:36.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:36.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:36.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:37.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:37.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:37.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:37.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:37.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:37.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:37.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:37.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:37.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:37.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:38.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:38.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:38.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:38.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:38.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:38.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:38.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:38.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:38.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:38.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:38.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:39.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:39.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:39.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:39.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:39.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:39.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:39.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:39.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:39.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:40.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:40.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:40.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:40.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:42.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:42.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:42.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:42.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:42.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:43.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:43.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:43.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:43.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:43.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:43.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:43.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:43.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:43.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:43.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:44.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:44.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:44.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:44.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:44.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:44.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:44.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:44.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:44.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:44.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:45.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:45.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:45.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:45.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:45.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:45.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:45.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:45.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:45.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:45.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:45.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:46.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:46.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:46.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:46.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:46.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:46.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:46.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:46.731 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-19 zones={test-zone2, test-zone1} 2021-02-11T00:34:46.731 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-19 zones={test-zone3, test-zone1} 2021-02-11T00:34:46.739 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:34:46.747 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:46.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.750 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:34:46.758 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:46.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:46.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:46.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:46.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:46.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:46.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:47.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:47.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:47.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:47.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:47.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:47.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:47.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:47.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:47.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:47.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:48.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:48.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:48.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:48.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:48.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:48.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:48.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:48.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:48.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:48.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:48.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:49.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:49.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:49.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:49.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:49.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:49.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:49.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:49.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:49.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:49.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:50.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:50.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:50.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:50.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:50.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:50.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:50.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:50.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:50.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:50.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:51.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:51.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:51.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:51.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:51.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:51.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:51.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:51.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:51.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:51.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:52.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:52.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:52.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:52.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:52.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:52.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:52.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:52.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:52.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:53.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:53.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:53.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:53.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:53.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:53.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:53.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:53.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:53.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:54.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:54.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:54.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:54.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:54.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:54.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:54.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:54.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:54.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:54.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:54.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:55.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:55.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:55.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:55.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:55.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:55.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:55.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:55.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:55.799 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-19 zones={test-zone3, test-zone1} 2021-02-11T00:34:55.799 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-19 zones={test-zone3, test-zone2} 2021-02-11T00:34:55.809 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:34:55.816 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:55.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.818 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:34:55.827 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-02-11T00:34:55.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:34:55.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:55.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:55.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-02-11T00:34:55.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:56.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:56.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:56.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:56.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:56.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:56.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:56.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:56.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:56.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:56.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:57.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:57.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:57.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:57.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:57.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:57.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:57.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:57.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:57.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:57.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:58.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:58.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:58.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:58.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:58.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:58.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:58.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:58.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:58.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:58.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:58.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:59.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:59.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:59.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:59.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:59.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:59.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:59.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:59.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:59.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:34:59.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:00.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:00.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:00.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:00.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:00.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:00.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:00.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:00.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:00.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:01.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:01.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:01.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:01.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:02.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:02.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:02.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:02.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:02.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:02.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:02.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:02.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:03.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:03.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:03.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:03.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:03.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:03.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:03.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:04.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:04.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:04.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:04.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:04.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:04.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:04.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:04.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:05.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:05.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:05.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:05.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:05.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:05.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:05.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:05.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:05.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:05.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:06.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:06.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:06.230 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-19 zones={test-zone3, test-zone2} 2021-02-11T00:35:06.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:06.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:06.230 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:06.271 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:06.271 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:06.527 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"00000000000000000000:00000000000000001044","last_update":"2021-02-11T00:34:45.579768Z"},{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T00:34:45.582829Z"},{"marker":"00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T00:34:23.574930Z"}] 2021-02-11T00:35:06.527 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T00:35:06.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:06.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:06.528 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:06.571 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:06.571 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:06.819 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7281982967011226472},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001044","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.579768Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.582829Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000000","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.574930Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T00:35:06.819 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T00:35:06.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:06.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:06.820 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:06.863 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:06.863 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:07.136 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"00000000000000000000:00000000000000001044","last_update":"2021-02-11T00:34:45.579768Z"},{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T00:34:45.582829Z"},{"marker":"00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T00:34:23.574930Z"}] 2021-02-11T00:35:07.136 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T00:35:07.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:07.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:07.137 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:07.222 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:07.222 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:07.459 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13309572567137725785},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001044","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.579768Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.582829Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000000","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.574930Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T00:35:07.460 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T00:35:07.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:07.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:07.460 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:07.545 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:07.545 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:07.804 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"00000000000000000000:00000000000000001044","last_update":"2021-02-11T00:34:45.579768Z"},{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T00:34:45.582829Z"},{"marker":"00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T00:34:23.574930Z"}] 2021-02-11T00:35:07.805 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T00:35:07.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:07.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:07.805 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:07.848 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:07.848 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:08.092 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4767706083908344750},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001044","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.579768Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.582829Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000000","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.574930Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T00:35:08.092 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T00:35:08.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:08.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:08.093 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:08.172 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:08.173 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:08.431 INFO:teuthology.orchestra.run.smithi095.stdout:[{"marker":"00000000000000000000:00000000000000001044","last_update":"2021-02-11T00:34:45.569671Z"},{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T00:34:45.572834Z"},{"marker":"00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T00:34:23.563592Z"}] 2021-02-11T00:35:08.431 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T00:35:08.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:08.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:08.432 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:08.477 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:08.477 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:08.736 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":5992573881530547290},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001044","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.569671Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.572834Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000000","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.563592Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T00:35:08.737 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T00:35:08.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:08.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:08.738 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:08.779 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:08.779 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:09.035 INFO:teuthology.orchestra.run.smithi095.stdout:[{"marker":"00000000000000000000:00000000000000001044","last_update":"2021-02-11T00:34:45.569671Z"},{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T00:34:45.572834Z"},{"marker":"00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T00:34:23.563592Z"}] 2021-02-11T00:35:09.036 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T00:35:09.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:09.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:09.037 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:09.080 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:09.080 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:09.334 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12481821032201219352},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001044","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.569671Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.572834Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000000","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.563592Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T00:35:09.334 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T00:35:09.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:09.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:09.335 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:09.415 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:09.416 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:09.692 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"00000000000000000000:00000000000000001044","last_update":"2021-02-11T00:34:45.545633Z"},{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T00:34:45.549594Z"},{"marker":"00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T00:34:23.539931Z"}] 2021-02-11T00:35:09.692 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T00:35:09.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:09.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:09.693 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:09.778 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:09.778 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:10.027 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15593642182000949264},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001044","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.545633Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.549594Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000000","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.539931Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T00:35:10.028 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T00:35:10.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:10.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:10.028 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:10.113 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:10.113 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:10.390 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"00000000000000000000:00000000000000001044","last_update":"2021-02-11T00:34:45.545633Z"},{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T00:34:45.549594Z"},{"marker":"00000000000000000000:00000000000000000000","last_update":"0.000000"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T00:34:23.539931Z"}] 2021-02-11T00:35:10.390 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T00:35:10.390 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:10.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:10.391 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:10.435 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:10.435 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:10.677 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11510982620117777593},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000001044","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.545633Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.549594Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000000","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.539931Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T00:35:10.677 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T00:35:10.678 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2021-02-11T00:35:10.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:10.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:10.721 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:10.759 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:10.759 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:10.952 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:35:10.992 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003582.559316_89.1","last_update":"2021-02-11T00:33:02.559316Z"},{"marker":"1_1613003710.715573_320.1","last_update":"2021-02-11T00:35:10.715573Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:35:10.992 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:35:10.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:10.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:10.993 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:11.037 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:11.038 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:11.260 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003710.715573_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:10.715573Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:11.260 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:35:11.260 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:35:11.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:11.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:11.261 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:11.303 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:11.303 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:11.548 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003710.715573_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:10.715573Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:11.548 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:35:11.548 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:35:11.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:11.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:11.549 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:11.590 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:11.591 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:11.828 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003710.715573_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:10.715573Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:11.829 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:35:11.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:11.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:11.879 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:11.926 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:11.927 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:12.141 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:12.136+0000 7fa103c8b080 1 waiting for incremental sync to catch up: 2021-02-11T00:35:12.141 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , , , ] 2021-02-11T00:35:12.141 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000001.27.5, , ] 2021-02-11T00:35:17.145 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:17.139+0000 7fa103c8b080 1 bucket sync caught up with source: 2021-02-11T00:35:17.145 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000001.27.5, , ] 2021-02-11T00:35:17.145 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000001.27.5, , ] 2021-02-11T00:35:17.145 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:17.139+0000 7fa103c8b080 0 bucket checkpoint complete 2021-02-11T00:35:17.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:17.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:17.181 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:17.227 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:17.228 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:17.441 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:17.440+0000 7f99bb384080 1 bucket sync caught up with source: 2021-02-11T00:35:17.441 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000001.27.5, , ] 2021-02-11T00:35:17.441 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000001.27.5, , ] 2021-02-11T00:35:17.441 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:17.440+0000 7f99bb384080 0 bucket checkpoint complete 2021-02-11T00:35:17.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:17.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:17.469 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:17.515 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:17.515 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:17.729 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:17.723+0000 7fcbee089080 1 bucket sync caught up with source: 2021-02-11T00:35:17.729 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000001.27.5, , ] 2021-02-11T00:35:17.729 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000001.27.5, , ] 2021-02-11T00:35:17.729 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:17.723+0000 7fcbee089080 0 bucket checkpoint complete 2021-02-11T00:35:17.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:17.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:17.762 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:17.810 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:17.810 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:18.026 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:18.024+0000 7f6702d03080 1 bucket sync caught up with source: 2021-02-11T00:35:18.026 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000001.30.5, , ] 2021-02-11T00:35:18.027 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000001.30.5, , ] 2021-02-11T00:35:18.027 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:18.024+0000 7f6702d03080 0 bucket checkpoint complete 2021-02-11T00:35:18.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:18.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:18.063 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:18.110 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:18.110 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:18.333 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:18.332+0000 7fcf6090f080 1 bucket sync caught up with source: 2021-02-11T00:35:18.333 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000001.30.5, , ] 2021-02-11T00:35:18.333 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000001.30.5, , ] 2021-02-11T00:35:18.333 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:18.332+0000 7fcf6090f080 0 bucket checkpoint complete 2021-02-11T00:35:18.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:18.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:18.368 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:18.460 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:18.460 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:18.694 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:18.692+0000 7fa1e9aaf080 1 bucket sync caught up with source: 2021-02-11T00:35:18.694 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000001.52.5, , ] 2021-02-11T00:35:18.694 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000001.52.5, , ] 2021-02-11T00:35:18.694 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:18.692+0000 7fa1e9aaf080 0 bucket checkpoint complete 2021-02-11T00:35:18.723 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:18.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:18.723 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:18.772 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:18.772 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:18.999 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:18.995+0000 7f244b941080 1 bucket sync caught up with source: 2021-02-11T00:35:19.000 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000001.52.5, , ] 2021-02-11T00:35:19.000 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000001.52.5, , ] 2021-02-11T00:35:19.000 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:18.995+0000 7f244b941080 0 bucket checkpoint complete 2021-02-11T00:35:19.037 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-20 zones={test-zone2, test-zone1} 2021-02-11T00:35:19.048 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:19.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:19.095 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:19.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:19.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:19.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:19.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:19.294 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-20 zones={test-zone2, test-zone1} 2021-02-11T00:35:19.294 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-20 zones={test-zone3, test-zone1} 2021-02-11T00:35:19.304 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:19.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:19.351 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:19.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:19.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:19.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:19.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:19.557 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-20 zones={test-zone3, test-zone1} 2021-02-11T00:35:19.558 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-20 zones={test-zone3, test-zone2} 2021-02-11T00:35:19.569 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:19.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:19.619 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:19.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:19.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:19.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:19.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:19.825 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-20 zones={test-zone3, test-zone2} 2021-02-11T00:35:19.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:19.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:19.842 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:19.880 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:19.880 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:20.068 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:35:20.101 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003582.559316_89.1","last_update":"2021-02-11T00:33:02.559316Z"},{"marker":"1_1613003719.838940_354.1","last_update":"2021-02-11T00:35:19.838940Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:35:20.101 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:35:20.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:20.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:20.102 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:20.139 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:20.139 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:20.379 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003719.838940_354.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:19.838940Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:20.379 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:35:20.380 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:35:20.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:20.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:20.380 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:20.422 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:20.422 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:20.673 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003719.838940_354.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:19.838940Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:20.673 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:35:20.673 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:35:20.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:20.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:20.674 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:20.714 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:20.714 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:20.945 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003719.838940_354.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:19.838940Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:20.945 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:35:20.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:20.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:20.995 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:21.040 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:21.040 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:21.245 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:21.243+0000 7ff39db3b080 1 bucket sync caught up with source: 2021-02-11T00:35:21.245 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000003.30.12, , ] 2021-02-11T00:35:21.245 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000003.30.12, , ] 2021-02-11T00:35:21.245 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:21.243+0000 7ff39db3b080 0 bucket checkpoint complete 2021-02-11T00:35:21.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:21.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:21.276 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:21.326 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:21.326 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:21.530 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:21.528+0000 7efe97288080 1 bucket sync caught up with source: 2021-02-11T00:35:21.979 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000003.30.12, , ] 2021-02-11T00:35:21.979 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000003.30.12, , ] 2021-02-11T00:35:21.979 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:21.528+0000 7efe97288080 0 bucket checkpoint complete 2021-02-11T00:35:21.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:21.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:21.981 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:22.030 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:22.031 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:22.225 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:22.223+0000 7f7ae5dc9080 1 bucket sync caught up with source: 2021-02-11T00:35:22.225 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000003.30.12, , ] 2021-02-11T00:35:22.225 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000003.30.12, , ] 2021-02-11T00:35:22.225 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:22.223+0000 7f7ae5dc9080 0 bucket checkpoint complete 2021-02-11T00:35:22.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:22.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:22.259 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:22.308 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:22.308 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:22.540 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:22.540+0000 7f08efae5080 1 bucket sync caught up with source: 2021-02-11T00:35:22.540 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000004.36.9, , ] 2021-02-11T00:35:22.540 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000004.36.9, , ] 2021-02-11T00:35:22.541 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:22.540+0000 7f08efae5080 0 bucket checkpoint complete 2021-02-11T00:35:22.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:22.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:22.578 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:22.622 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:22.622 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:22.853 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:22.852+0000 7effe47d4080 1 bucket sync caught up with source: 2021-02-11T00:35:22.853 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000004.36.9, , ] 2021-02-11T00:35:22.853 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000004.36.9, , ] 2021-02-11T00:35:22.853 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:22.852+0000 7effe47d4080 0 bucket checkpoint complete 2021-02-11T00:35:22.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:22.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:22.889 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:22.980 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:22.980 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:23.221 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:23.220+0000 7f42f0dfe080 1 bucket sync caught up with source: 2021-02-11T00:35:23.222 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000004.58.9, , ] 2021-02-11T00:35:23.222 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000004.58.9, , ] 2021-02-11T00:35:23.222 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:23.220+0000 7f42f0dfe080 0 bucket checkpoint complete 2021-02-11T00:35:23.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:23.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:23.251 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:23.298 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:23.298 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:23.507 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:23.507+0000 7f8222ec4080 1 bucket sync caught up with source: 2021-02-11T00:35:23.507 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000004.58.9, , ] 2021-02-11T00:35:23.507 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000004.58.9, , ] 2021-02-11T00:35:23.507 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:23.507+0000 7f8222ec4080 0 bucket checkpoint complete 2021-02-11T00:35:23.542 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-20 zones={test-zone2, test-zone1} 2021-02-11T00:35:23.553 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:23.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:23.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:23.603 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:23.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:23.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:23.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:23.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:23.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:23.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:23.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:23.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:23.947 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-20 zones={test-zone2, test-zone1} 2021-02-11T00:35:23.947 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-20 zones={test-zone3, test-zone1} 2021-02-11T00:35:23.954 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:23.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:23.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:23.999 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:24.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:24.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:24.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:24.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:24.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:24.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:24.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:24.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:24.329 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-20 zones={test-zone3, test-zone1} 2021-02-11T00:35:24.329 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-20 zones={test-zone3, test-zone2} 2021-02-11T00:35:24.340 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:24.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:24.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:24.391 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:24.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:24.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:24.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:24.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:24.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:24.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:24.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:24.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:26.259 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-20 zones={test-zone3, test-zone2} 2021-02-11T00:35:26.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:26.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:26.275 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:26.313 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:26.314 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:26.509 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:35:26.546 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003582.559316_89.1","last_update":"2021-02-11T00:33:02.559316Z"},{"marker":"1_1613003726.272815_400.1","last_update":"2021-02-11T00:35:26.272815Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:35:26.546 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:35:26.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:26.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:26.547 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:26.586 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:26.586 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:26.823 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003726.272815_400.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:26.272815Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:26.823 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:35:26.823 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:35:26.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:26.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:26.824 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:26.869 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:26.869 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:27.109 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003726.272815_400.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:26.272815Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:27.110 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:35:27.110 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:35:27.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:27.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:27.111 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:27.151 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:27.151 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:27.371 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003726.272815_400.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:26.272815Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:27.372 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:35:27.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:27.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:27.427 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:27.476 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:27.476 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:27.687 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:27.687+0000 7f9b3acaf080 1 bucket sync caught up with source: 2021-02-11T00:35:27.687 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000005.34.17, , ] 2021-02-11T00:35:27.687 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000005.34.17, , ] 2021-02-11T00:35:27.687 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:27.687+0000 7f9b3acaf080 0 bucket checkpoint complete 2021-02-11T00:35:27.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:27.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:27.720 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:27.771 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:27.771 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:27.983 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:27.980+0000 7fcc8b434080 1 bucket sync caught up with source: 2021-02-11T00:35:27.983 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000005.34.17, , ] 2021-02-11T00:35:27.983 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000005.34.17, , ] 2021-02-11T00:35:27.984 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:27.980+0000 7fcc8b434080 0 bucket checkpoint complete 2021-02-11T00:35:28.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:28.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:28.012 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:28.059 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:28.059 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:28.265 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:28.263+0000 7fee53574080 1 bucket sync caught up with source: 2021-02-11T00:35:28.265 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000005.34.17, , ] 2021-02-11T00:35:28.265 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000005.34.17, , ] 2021-02-11T00:35:28.265 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:28.263+0000 7fee53574080 0 bucket checkpoint complete 2021-02-11T00:35:28.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:28.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:28.298 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:28.348 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:28.348 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:28.578 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:28.576+0000 7fa84fdb1080 1 bucket sync caught up with source: 2021-02-11T00:35:28.578 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000007.43.14, , ] 2021-02-11T00:35:28.578 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000007.43.14, , ] 2021-02-11T00:35:28.578 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:28.576+0000 7fa84fdb1080 0 bucket checkpoint complete 2021-02-11T00:35:28.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:28.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:28.614 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:28.662 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:28.662 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:28.884 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:28.880+0000 7fe43c8de080 1 bucket sync caught up with source: 2021-02-11T00:35:28.884 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000007.43.14, , ] 2021-02-11T00:35:28.884 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000007.43.14, , ] 2021-02-11T00:35:28.884 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:28.880+0000 7fe43c8de080 0 bucket checkpoint complete 2021-02-11T00:35:28.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:28.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:28.918 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:28.965 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:28.965 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:29.216 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:29.212+0000 7fabe64a5080 1 bucket sync caught up with source: 2021-02-11T00:35:29.216 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000007.65.14, , ] 2021-02-11T00:35:29.216 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000007.65.14, , ] 2021-02-11T00:35:29.216 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:29.212+0000 7fabe64a5080 0 bucket checkpoint complete 2021-02-11T00:35:29.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:29.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:29.246 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:29.294 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:29.295 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:29.516 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:29.515+0000 7f13435b5080 1 bucket sync caught up with source: 2021-02-11T00:35:29.516 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000007.65.14, , ] 2021-02-11T00:35:29.516 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000007.65.14, , ] 2021-02-11T00:35:29.516 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:29.515+0000 7f13435b5080 0 bucket checkpoint complete 2021-02-11T00:35:29.550 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-20 zones={test-zone2, test-zone1} 2021-02-11T00:35:29.560 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:29.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:29.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:29.607 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:29.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:29.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:29.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:29.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:29.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:29.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:29.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:29.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:29.945 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-20 zones={test-zone2, test-zone1} 2021-02-11T00:35:29.946 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-20 zones={test-zone3, test-zone1} 2021-02-11T00:35:29.956 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:30.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:30.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:30.003 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:30.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:30.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:30.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:30.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:30.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:30.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:30.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:30.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:30.341 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-20 zones={test-zone3, test-zone1} 2021-02-11T00:35:30.342 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-20 zones={test-zone3, test-zone2} 2021-02-11T00:35:30.353 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:30.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:30.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:30.399 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:30.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:30.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:30.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:30.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:30.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:30.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:30.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:30.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:30.742 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-20 zones={test-zone3, test-zone2} 2021-02-11T00:35:30.742 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2021-02-11T00:35:30.743 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-21 2021-02-11T00:35:30.789 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-22 2021-02-11T00:35:30.842 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-23 2021-02-11T00:35:30.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:30.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:30.944 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:30.979 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:30.979 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:31.175 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:35:31.215 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003582.559316_89.1","last_update":"2021-02-11T00:33:02.559316Z"},{"marker":"1_1613003730.936764_462.1","last_update":"2021-02-11T00:35:30.936764Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:35:31.215 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:35:31.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:31.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:31.216 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:31.256 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:31.256 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:31.498 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003730.936764_462.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:30.936764Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:31.498 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:35:31.498 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:35:31.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:31.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:31.499 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:31.539 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:31.539 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:31.782 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003730.936764_462.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:30.936764Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:31.783 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:35:31.783 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:35:31.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:31.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:31.783 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:31.824 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:31.824 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:32.053 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003730.936764_462.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:30.936764Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:32.054 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:35:32.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:32.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:32.276 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:32.325 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:32.325 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:32.530 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:32.531+0000 7fd013bb9080 1 bucket sync caught up with source: 2021-02-11T00:35:33.989 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000003.20.11, , ] 2021-02-11T00:35:33.989 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000003.20.11, , ] 2021-02-11T00:35:33.989 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:32.531+0000 7fd013bb9080 0 bucket checkpoint complete 2021-02-11T00:35:33.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:33.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:33.991 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:34.039 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:34.039 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:34.279 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:34.276+0000 7f4a7e18b080 1 bucket sync caught up with source: 2021-02-11T00:35:34.280 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000003.20.11, , ] 2021-02-11T00:35:34.280 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000003.20.11, , ] 2021-02-11T00:35:34.280 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:34.276+0000 7f4a7e18b080 0 bucket checkpoint complete 2021-02-11T00:35:34.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:34.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:34.311 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:34.360 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:34.404 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:34.586 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:34.582+0000 7f33cee24080 1 bucket sync caught up with source: 2021-02-11T00:35:34.586 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000003.20.11, , ] 2021-02-11T00:35:34.586 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000003.20.11, , ] 2021-02-11T00:35:34.586 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:34.582+0000 7f33cee24080 0 bucket checkpoint complete 2021-02-11T00:35:34.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:34.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:34.620 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:34.668 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:34.668 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:34.895 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:34.892+0000 7efcfbee2080 1 bucket sync caught up with source: 2021-02-11T00:35:34.896 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000002.41.5, , ] 2021-02-11T00:35:34.896 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000002.41.5, , ] 2021-02-11T00:35:34.896 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:34.892+0000 7efcfbee2080 0 bucket checkpoint complete 2021-02-11T00:35:34.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:34.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:34.933 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:34.980 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:34.980 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:35.225 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:35.224+0000 7f58692ff080 1 bucket sync caught up with source: 2021-02-11T00:35:35.339 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000002.41.5, , ] 2021-02-11T00:35:35.339 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000002.41.5, , ] 2021-02-11T00:35:35.339 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:35.224+0000 7f58692ff080 0 bucket checkpoint complete 2021-02-11T00:35:35.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:35.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:35.341 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:35.389 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:35.389 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:35.633 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:35.631+0000 7f066aa6b080 1 bucket sync caught up with source: 2021-02-11T00:35:35.633 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000002.24.5, , ] 2021-02-11T00:35:35.633 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000002.24.5, , ] 2021-02-11T00:35:35.633 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:35.631+0000 7f066aa6b080 0 bucket checkpoint complete 2021-02-11T00:35:35.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:35.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:35.666 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:35.712 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:35.712 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:35.934 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:35.930+0000 7fd6c9e3e080 1 bucket sync caught up with source: 2021-02-11T00:35:35.934 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000002.24.5, , ] 2021-02-11T00:35:35.934 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000002.24.5, , ] 2021-02-11T00:35:35.934 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:35.930+0000 7fd6c9e3e080 0 bucket checkpoint complete 2021-02-11T00:35:35.966 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-21 zones={test-zone2, test-zone1} 2021-02-11T00:35:35.979 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:36.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:36.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:36.042 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:36.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:36.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:36.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:36.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:36.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:36.237 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-21 zones={test-zone2, test-zone1} 2021-02-11T00:35:36.237 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-21 zones={test-zone3, test-zone1} 2021-02-11T00:35:36.250 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:36.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:36.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:36.300 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:36.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:36.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:36.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:36.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:36.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:36.496 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-21 zones={test-zone3, test-zone1} 2021-02-11T00:35:36.497 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-21 zones={test-zone3, test-zone2} 2021-02-11T00:35:36.510 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:36.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:36.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:36.560 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:36.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:36.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:36.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:36.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:36.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:36.727 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-21 zones={test-zone3, test-zone2} 2021-02-11T00:35:36.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:36.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:36.727 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:36.773 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:36.773 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:37.010 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:37.006+0000 7f0be5a12080 1 bucket sync caught up with source: 2021-02-11T00:35:37.010 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000002.14.5, , ] 2021-02-11T00:35:37.010 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000002.14.5, , ] 2021-02-11T00:35:37.010 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:37.006+0000 7f0be5a12080 0 bucket checkpoint complete 2021-02-11T00:35:37.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:37.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:37.037 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:37.085 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:37.085 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:37.311 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:37.307+0000 7f65870e1080 1 bucket sync caught up with source: 2021-02-11T00:35:37.311 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000002.14.5, , ] 2021-02-11T00:35:37.311 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000002.14.5, , ] 2021-02-11T00:35:37.311 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:37.307+0000 7f65870e1080 0 bucket checkpoint complete 2021-02-11T00:35:37.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:37.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:37.341 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:37.388 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:37.388 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:37.603 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:37.602+0000 7f8ce4c28080 1 bucket sync caught up with source: 2021-02-11T00:35:37.603 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000002.14.5, , ] 2021-02-11T00:35:37.603 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000002.14.5, , ] 2021-02-11T00:35:37.604 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:37.602+0000 7f8ce4c28080 0 bucket checkpoint complete 2021-02-11T00:35:37.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:37.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:37.638 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:37.685 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:37.685 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:37.928 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:37.927+0000 7f07b4454080 1 bucket sync caught up with source: 2021-02-11T00:35:37.929 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000003.21.11, , ] 2021-02-11T00:35:37.929 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000003.21.11, , ] 2021-02-11T00:35:37.929 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:37.927+0000 7f07b4454080 0 bucket checkpoint complete 2021-02-11T00:35:37.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:37.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:37.954 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:37.997 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:37.997 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:38.227 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:38.223+0000 7f4fbab4e080 1 bucket sync caught up with source: 2021-02-11T00:35:38.227 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000003.21.11, , ] 2021-02-11T00:35:38.227 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000003.21.11, , ] 2021-02-11T00:35:38.228 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:38.223+0000 7f4fbab4e080 0 bucket checkpoint complete 2021-02-11T00:35:38.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:38.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:38.263 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:38.352 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:38.352 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:38.580 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:38.575+0000 7f68e1e2a080 1 bucket sync caught up with source: 2021-02-11T00:35:38.580 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000002.23.5, , ] 2021-02-11T00:35:38.580 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000002.23.5, , ] 2021-02-11T00:35:38.580 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:38.575+0000 7f68e1e2a080 0 bucket checkpoint complete 2021-02-11T00:35:38.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:38.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:38.611 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:38.660 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:38.660 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:38.887 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:38.886+0000 7fedaa562080 1 bucket sync caught up with source: 2021-02-11T00:35:38.887 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000002.23.5, , ] 2021-02-11T00:35:38.887 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000002.23.5, , ] 2021-02-11T00:35:38.888 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:38.886+0000 7fedaa562080 0 bucket checkpoint complete 2021-02-11T00:35:38.922 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-22 zones={test-zone2, test-zone1} 2021-02-11T00:35:38.933 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:38.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:38.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:38.987 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:38.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:38.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:39.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:39.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:39.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:39.140 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-22 zones={test-zone2, test-zone1} 2021-02-11T00:35:39.140 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-22 zones={test-zone3, test-zone1} 2021-02-11T00:35:39.148 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:39.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:39.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:39.195 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:39.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:39.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:39.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:39.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:39.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:39.350 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-22 zones={test-zone3, test-zone1} 2021-02-11T00:35:39.350 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-22 zones={test-zone3, test-zone2} 2021-02-11T00:35:39.359 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:39.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:39.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:39.407 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:39.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:39.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:39.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:39.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:39.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:39.557 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-22 zones={test-zone3, test-zone2} 2021-02-11T00:35:39.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:39.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:39.557 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:39.602 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:39.602 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:39.813 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:39.810+0000 7f3a1d40c080 1 bucket sync caught up with source: 2021-02-11T00:35:39.813 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000002.26.5, , ] 2021-02-11T00:35:39.814 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000002.26.5, , ] 2021-02-11T00:35:39.814 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:39.810+0000 7f3a1d40c080 0 bucket checkpoint complete 2021-02-11T00:35:39.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:39.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:39.846 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:39.894 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:39.894 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:40.121 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:40.119+0000 7fcaa7845080 1 bucket sync caught up with source: 2021-02-11T00:35:40.122 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000002.26.5, , ] 2021-02-11T00:35:40.122 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000002.26.5, , ] 2021-02-11T00:35:40.122 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:40.119+0000 7fcaa7845080 0 bucket checkpoint complete 2021-02-11T00:35:40.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:40.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:40.150 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:40.196 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:40.196 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:40.421 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:40.418+0000 7f403dd3f080 1 bucket sync caught up with source: 2021-02-11T00:35:40.421 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000002.26.5, , ] 2021-02-11T00:35:40.421 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000002.26.5, , ] 2021-02-11T00:35:40.421 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:40.418+0000 7f403dd3f080 0 bucket checkpoint complete 2021-02-11T00:35:40.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:40.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:40.454 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:40.503 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:40.503 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:40.756 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:40.751+0000 7fdcda996080 1 bucket sync caught up with source: 2021-02-11T00:35:40.756 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000002.54.5, , ] 2021-02-11T00:35:40.756 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000002.54.5, , ] 2021-02-11T00:35:40.756 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:40.751+0000 7fdcda996080 0 bucket checkpoint complete 2021-02-11T00:35:40.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:40.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:40.792 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:40.837 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:40.837 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:41.073 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:41.071+0000 7f5f43527080 1 bucket sync caught up with source: 2021-02-11T00:35:41.074 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000002.54.5, , ] 2021-02-11T00:35:41.074 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000002.54.5, , ] 2021-02-11T00:35:41.074 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:41.071+0000 7f5f43527080 0 bucket checkpoint complete 2021-02-11T00:35:41.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:41.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:41.111 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:41.156 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:41.156 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:41.387 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:41.383+0000 7f5bdf343080 1 bucket sync caught up with source: 2021-02-11T00:35:41.387 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000003.41.11, , ] 2021-02-11T00:35:41.387 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000003.41.11, , ] 2021-02-11T00:35:41.387 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:41.383+0000 7f5bdf343080 0 bucket checkpoint complete 2021-02-11T00:35:41.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:41.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:41.421 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:41.469 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:41.469 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:41.682 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:41.678+0000 7f69a565d080 1 bucket sync caught up with source: 2021-02-11T00:35:41.682 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000003.41.11, , ] 2021-02-11T00:35:41.682 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000003.41.11, , ] 2021-02-11T00:35:41.682 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:41.678+0000 7f69a565d080 0 bucket checkpoint complete 2021-02-11T00:35:41.715 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-23 zones={test-zone2, test-zone1} 2021-02-11T00:35:41.726 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:41.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:41.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:41.775 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:41.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:41.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:41.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:41.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:41.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:41.924 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-23 zones={test-zone2, test-zone1} 2021-02-11T00:35:41.925 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-23 zones={test-zone3, test-zone1} 2021-02-11T00:35:41.932 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:41.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:41.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:41.979 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:41.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:41.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:42.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:42.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:42.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:42.129 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-23 zones={test-zone3, test-zone1} 2021-02-11T00:35:42.129 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-23 zones={test-zone3, test-zone2} 2021-02-11T00:35:42.137 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:42.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:42.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:42.183 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:42.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:42.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:42.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:42.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:42.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:42.337 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-23 zones={test-zone3, test-zone2} 2021-02-11T00:35:42.337 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2021-02-11T00:35:42.338 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-24 2021-02-11T00:35:42.380 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-25 2021-02-11T00:35:42.434 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-26 2021-02-11T00:35:42.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:42.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:42.591 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:42.630 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:42.631 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:42.817 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:35:42.855 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003582.559316_89.1","last_update":"2021-02-11T00:33:02.559316Z"},{"marker":"1_1613003742.584739_493.1","last_update":"2021-02-11T00:35:42.584739Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:35:42.855 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:35:42.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:42.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:42.856 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:42.897 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:42.897 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:43.113 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003742.584739_493.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:42.584739Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:43.113 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:35:43.114 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:35:43.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:43.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:43.114 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:43.157 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:43.157 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:43.407 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003742.584739_493.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:42.584739Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:43.407 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:35:43.407 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:35:43.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:43.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:43.408 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:43.449 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:43.449 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:43.676 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003742.584739_493.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:42.584739Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:43.676 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:35:43.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:44.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:44.037 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:44.088 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:44.088 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:44.300 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:44.298+0000 7f14203e1080 1 bucket sync caught up with source: 2021-02-11T00:35:44.301 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000003.12.6, , ] 2021-02-11T00:35:44.301 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000003.12.6, , ] 2021-02-11T00:35:44.301 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:44.298+0000 7f14203e1080 0 bucket checkpoint complete 2021-02-11T00:35:44.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:44.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:44.336 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:44.385 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:44.386 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:44.600 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:44.595+0000 7fe2eaa1d080 1 bucket sync caught up with source: 2021-02-11T00:35:44.600 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000003.12.6, , ] 2021-02-11T00:35:44.600 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000003.12.6, , ] 2021-02-11T00:35:44.600 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:44.595+0000 7fe2eaa1d080 0 bucket checkpoint complete 2021-02-11T00:35:44.630 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:44.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:44.631 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:44.680 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:44.680 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:44.870 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:44.866+0000 7fedea654080 1 bucket sync caught up with source: 2021-02-11T00:35:45.251 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000003.12.6, , ] 2021-02-11T00:35:45.251 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000003.12.6, , ] 2021-02-11T00:35:45.251 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:44.866+0000 7fedea654080 0 bucket checkpoint complete 2021-02-11T00:35:45.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:45.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:45.253 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:45.301 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:45.301 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:45.529 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:45.527+0000 7f3c7a2a7080 1 bucket sync caught up with source: 2021-02-11T00:35:45.621 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000001.13.12, , ] 2021-02-11T00:35:45.621 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000001.13.12, , ] 2021-02-11T00:35:45.621 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:45.527+0000 7f3c7a2a7080 0 bucket checkpoint complete 2021-02-11T00:35:45.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:45.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:45.623 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:45.669 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:45.669 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:45.904 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:45.903+0000 7ff574bb3080 1 bucket sync caught up with source: 2021-02-11T00:35:45.905 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000001.13.12, , ] 2021-02-11T00:35:45.905 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000001.13.12, , ] 2021-02-11T00:35:45.905 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:45.903+0000 7ff574bb3080 0 bucket checkpoint complete 2021-02-11T00:35:45.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:45.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:45.939 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:45.987 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:45.988 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:46.217 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:46.215+0000 7f207de77080 1 bucket sync caught up with source: 2021-02-11T00:35:46.217 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000001.9.12, , ] 2021-02-11T00:35:46.217 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000001.9.12, , ] 2021-02-11T00:35:46.217 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:46.215+0000 7f207de77080 0 bucket checkpoint complete 2021-02-11T00:35:46.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:46.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:46.248 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:46.295 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:46.296 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:46.545 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:46.542+0000 7fd895f77080 1 bucket sync caught up with source: 2021-02-11T00:35:46.545 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000001.9.12, , ] 2021-02-11T00:35:46.545 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000001.9.12, , ] 2021-02-11T00:35:46.545 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:46.542+0000 7fd895f77080 0 bucket checkpoint complete 2021-02-11T00:35:46.580 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-24 zones={test-zone2, test-zone1} 2021-02-11T00:35:46.600 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:46.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:46.652 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:46.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:46.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:46.742 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-24 zones={test-zone2, test-zone1} 2021-02-11T00:35:46.743 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-24 zones={test-zone3, test-zone1} 2021-02-11T00:35:46.762 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:46.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:46.823 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:46.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:46.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:46.906 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-24 zones={test-zone3, test-zone1} 2021-02-11T00:35:46.906 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-24 zones={test-zone3, test-zone2} 2021-02-11T00:35:46.923 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:46.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:46.976 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:46.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:47.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:47.051 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-24 zones={test-zone3, test-zone2} 2021-02-11T00:35:47.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:47.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:47.052 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:47.098 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:47.098 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:47.325 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:47.321+0000 7f0f57ecd080 1 bucket sync caught up with source: 2021-02-11T00:35:47.326 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000001.9.12, , ] 2021-02-11T00:35:47.326 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000001.9.12, , ] 2021-02-11T00:35:47.326 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:47.321+0000 7f0f57ecd080 0 bucket checkpoint complete 2021-02-11T00:35:47.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:47.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:47.365 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:47.413 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:47.414 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:47.635 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:47.631+0000 7f85ef9be080 1 bucket sync caught up with source: 2021-02-11T00:35:47.635 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000001.9.12, , ] 2021-02-11T00:35:47.635 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000001.9.12, , ] 2021-02-11T00:35:47.635 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:47.631+0000 7f85ef9be080 0 bucket checkpoint complete 2021-02-11T00:35:47.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:47.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:47.666 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:47.712 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:47.712 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:47.928 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:47.925+0000 7f172cbbe080 1 bucket sync caught up with source: 2021-02-11T00:35:47.928 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000001.9.12, , ] 2021-02-11T00:35:47.928 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000001.9.12, , ] 2021-02-11T00:35:47.928 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:47.925+0000 7f172cbbe080 0 bucket checkpoint complete 2021-02-11T00:35:47.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:47.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:47.959 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:48.006 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:48.006 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:48.250 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:48.247+0000 7f6edc143080 1 bucket sync caught up with source: 2021-02-11T00:35:48.251 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000003.21.6, , ] 2021-02-11T00:35:48.251 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000003.21.6, , ] 2021-02-11T00:35:48.251 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:48.247+0000 7f6edc143080 0 bucket checkpoint complete 2021-02-11T00:35:48.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:48.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:48.285 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:48.332 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:48.332 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:48.557 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:48.555+0000 7fd5a30a4080 1 bucket sync caught up with source: 2021-02-11T00:35:48.557 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000003.21.6, , ] 2021-02-11T00:35:48.557 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000003.21.6, , ] 2021-02-11T00:35:48.558 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:48.555+0000 7fd5a30a4080 0 bucket checkpoint complete 2021-02-11T00:35:48.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:48.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:48.593 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:48.640 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:48.640 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:48.868 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:48.863+0000 7f16bcf50080 1 bucket sync caught up with source: 2021-02-11T00:35:48.868 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000001.16.12, , ] 2021-02-11T00:35:48.868 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000001.16.12, , ] 2021-02-11T00:35:48.868 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:48.863+0000 7f16bcf50080 0 bucket checkpoint complete 2021-02-11T00:35:48.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:48.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:48.902 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:48.949 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:48.949 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:49.186 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:49.181+0000 7f2075ad8080 1 bucket sync caught up with source: 2021-02-11T00:35:49.186 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000001.16.12, , ] 2021-02-11T00:35:49.186 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000001.16.12, , ] 2021-02-11T00:35:49.186 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:49.181+0000 7f2075ad8080 0 bucket checkpoint complete 2021-02-11T00:35:49.222 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-25 zones={test-zone2, test-zone1} 2021-02-11T00:35:49.231 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:49.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:49.279 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:49.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:49.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:49.346 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-25 zones={test-zone2, test-zone1} 2021-02-11T00:35:49.351 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-25 zones={test-zone3, test-zone1} 2021-02-11T00:35:49.358 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:49.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:49.407 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:49.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:49.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:49.472 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-25 zones={test-zone3, test-zone1} 2021-02-11T00:35:49.472 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-25 zones={test-zone3, test-zone2} 2021-02-11T00:35:49.482 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:49.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:49.527 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:49.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:49.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:49.596 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-25 zones={test-zone3, test-zone2} 2021-02-11T00:35:49.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:49.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:49.596 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:49.642 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:49.643 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:49.860 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:49.857+0000 7f15a1a0b080 1 bucket sync caught up with source: 2021-02-11T00:35:49.861 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000001.46.12, , ] 2021-02-11T00:35:49.861 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000001.46.12, , ] 2021-02-11T00:35:49.861 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:49.857+0000 7f15a1a0b080 0 bucket checkpoint complete 2021-02-11T00:35:49.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:49.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:49.895 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:49.936 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:49.936 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:50.136 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:50.135+0000 7fa86a1ff080 1 bucket sync caught up with source: 2021-02-11T00:35:50.137 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000001.46.12, , ] 2021-02-11T00:35:50.137 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000001.46.12, , ] 2021-02-11T00:35:50.137 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:50.135+0000 7fa86a1ff080 0 bucket checkpoint complete 2021-02-11T00:35:50.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:50.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:50.167 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:50.216 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:50.217 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:50.437 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:50.433+0000 7f3e79dfb080 1 bucket sync caught up with source: 2021-02-11T00:35:50.437 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000001.46.12, , ] 2021-02-11T00:35:50.437 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000001.46.12, , ] 2021-02-11T00:35:50.437 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:50.433+0000 7f3e79dfb080 0 bucket checkpoint complete 2021-02-11T00:35:50.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:50.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:50.474 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:50.522 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:50.522 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:50.762 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:50.759+0000 7f0c922dd080 1 bucket sync caught up with source: 2021-02-11T00:35:50.762 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000001.40.12, , ] 2021-02-11T00:35:50.763 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000001.40.12, , ] 2021-02-11T00:35:50.763 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:50.759+0000 7f0c922dd080 0 bucket checkpoint complete 2021-02-11T00:35:50.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:50.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:50.799 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:50.849 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:50.849 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:51.065 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:51.063+0000 7fd1b0b38080 1 bucket sync caught up with source: 2021-02-11T00:35:51.065 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000001.40.12, , ] 2021-02-11T00:35:51.065 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000001.40.12, , ] 2021-02-11T00:35:51.065 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:51.063+0000 7fd1b0b38080 0 bucket checkpoint complete 2021-02-11T00:35:51.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:51.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:51.102 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:51.149 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:51.149 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:51.386 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:51.383+0000 7f02113c5080 1 bucket sync caught up with source: 2021-02-11T00:35:51.386 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , , , , 00000000003.48.6, , ] 2021-02-11T00:35:51.387 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , , , , 00000000003.48.6, , ] 2021-02-11T00:35:51.387 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:35:51.383+0000 7f02113c5080 0 bucket checkpoint complete 2021-02-11T00:35:51.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:51.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T00:35:51.418 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T00:35:51.466 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:51.466 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:51.691 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:51.685+0000 7f3de0b60080 1 bucket sync caught up with source: 2021-02-11T00:35:51.691 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , , , , 00000000003.48.6, , ] 2021-02-11T00:35:51.691 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , , , , 00000000003.48.6, , ] 2021-02-11T00:35:51.691 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:35:51.685+0000 7f3de0b60080 0 bucket checkpoint complete 2021-02-11T00:35:51.723 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-26 zones={test-zone2, test-zone1} 2021-02-11T00:35:51.733 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:51.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:51.779 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:51.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:51.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:51.846 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-26 zones={test-zone2, test-zone1} 2021-02-11T00:35:51.847 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-26 zones={test-zone3, test-zone1} 2021-02-11T00:35:51.855 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:51.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:51.902 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:51.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:51.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:51.967 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-26 zones={test-zone3, test-zone1} 2021-02-11T00:35:51.967 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-26 zones={test-zone3, test-zone2} 2021-02-11T00:35:51.977 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:35:52.022 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:52.023 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:35:52.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-02-11T00:35:52.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-02-11T00:35:52.095 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-26 zones={test-zone3, test-zone2} 2021-02-11T00:35:52.095 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2021-02-11T00:35:52.096 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-27 2021-02-11T00:35:52.131 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-28 2021-02-11T00:35:52.187 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-29 2021-02-11T00:35:52.450 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2021-02-11T00:35:52.451 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-30 2021-02-11T00:35:52.484 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-31 2021-02-11T00:35:52.539 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-32 2021-02-11T00:35:52.787 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2021-02-11T00:35:52.787 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-33 2021-02-11T00:35:52.837 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-34 2021-02-11T00:35:52.891 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-35 2021-02-11T00:35:55.147 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2021-02-11T00:35:55.147 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-36 2021-02-11T00:35:55.191 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-37 2021-02-11T00:35:55.259 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-38 2021-02-11T00:35:55.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:55.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:55.313 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:55.349 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:55.349 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:55.529 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:35:55.568 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003582.559316_89.1","last_update":"2021-02-11T00:33:02.559316Z"},{"marker":"1_1613003755.291866_566.1","last_update":"2021-02-11T00:35:55.291866Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:35:55.569 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:35:55.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:55.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:55.569 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:55.609 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:55.609 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:55.835 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003755.291866_566.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:55.291866Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:56.824 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:35:56.824 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:35:56.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:56.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:56.825 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:56.866 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:56.866 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:57.102 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003755.291866_566.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:55.291866Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:57.102 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:35:57.102 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:35:57.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:57.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:57.103 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:57.166 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:57.167 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:57.393 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003755.291866_566.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:55.291866Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:57.394 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:35:57.770 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2021-02-11T00:35:57.770 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-39 2021-02-11T00:35:57.807 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-40 2021-02-11T00:35:57.871 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-41 2021-02-11T00:35:57.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:57.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:57.932 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:57.968 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:57.968 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:58.125 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=7f3547bf-73e3-43a7-aa91-648db0f982f6 2021-02-11T00:35:58.154 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"1_1613003582.559316_89.1","last_update":"2021-02-11T00:33:02.559316Z"},{"marker":"1_1613003757.913330_581.1","last_update":"2021-02-11T00:35:57.913330Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:35:58.154 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:35:58.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:58.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:58.155 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:58.196 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:58.196 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:58.432 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003757.913330_581.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:57.913330Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:58.433 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:35:58.433 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:35:58.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:58.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:58.433 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:58.474 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:58.474 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:58.727 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003757.913330_581.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:57.913330Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:58.727 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:35:58.727 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:35:58.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:58.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:35:58.728 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:35:58.771 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:35:58.771 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:35:59.008 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"7f3547bf-73e3-43a7-aa91-648db0f982f6","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003757.913330_581.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:57.913330Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:35:59.008 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:35:59.008 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-02-11T00:35:59.009 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T00:35:59.010 INFO:tasks.rgw.c1.client.1.smithi057.stdout:2021-02-11T00:35:59.007+0000 7f2d98f71700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 19837) UID: 0 2021-02-11T00:35:59.010 INFO:tasks.rgw.c1.client.1.smithi057.stdout:2021-02-11T00:35:59.007+0000 7f2db0acc0c0 -1 shutting down 2021-02-11T00:36:03.142 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-02-11T00:36:05.110 INFO:tasks.rgw.c1.client.1:Stopped 2021-02-11T00:36:05.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:36:05.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:36:05.111 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:36:05.151 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:36:05.151 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:36:05.177 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:36:05.173+0000 7fe3e746c080 0 NOTICE: overriding master zone: e989c9f2-9c89-43f5-a54f-e1d0705a9abe 2021-02-11T00:36:05.178 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:36:05.173+0000 7fe3e746c080 0 failed reading obj info from .rgw.root:zone_info.35e954f9-5ea0-411e-8e41-e821d2787b31: (2) No such file or directory 2021-02-11T00:36:05.178 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:36:05.173+0000 7fe3e746c080 0 WARNING: could not read zone params for zone id=35e954f9-5ea0-411e-8e41-e821d2787b31 name=test-zone3 2021-02-11T00:36:05.179 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:36:05.173+0000 7fe3e746c080 0 failed reading obj info from .rgw.root:zone_info.e989c9f2-9c89-43f5-a54f-e1d0705a9abe: (2) No such file or directory 2021-02-11T00:36:05.179 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:36:05.173+0000 7fe3e746c080 0 WARNING: could not read zone params for zone id=e989c9f2-9c89-43f5-a54f-e1d0705a9abe name=test-zone1 2021-02-11T00:36:05.191 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"CP2FO0ORQU6DSQKV","secret_key":"szUDueomrwAKvai4rUGz9pCzuDdncCWD"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T00:36:05.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:36:05.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:36:05.192 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:36:05.273 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:36:05.273 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:36:05.588 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"4d4a6517-ce7a-42dd-9c62-65c684a638fb","epoch":1,"predecessor_uuid":"7f3547bf-73e3-43a7-aa91-648db0f982f6","sync_status":["1_1613003582.559316_89.1","1_1613003757.913330_581.1","",""],"period_map":{"id":"4d4a6517-ce7a-42dd-9c62-65c684a638fb","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","zones":[{"id":"35e954f9-5ea0-411e-8e41-e821d2787b31","name":"test-zone3","endpoints":["http://smithi057.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","name":"test-zone4","endpoints":["http://smithi095.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","endpoints":["http://smithi095.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"35e954f9-5ea0-411e-8e41-e821d2787b31","val":398828966},{"key":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","val":1248431312},{"key":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","val":3133685257},{"key":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","val":1241439546}]},"master_zonegroup":"test-zonegroup","master_zone":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","realm_name":"test-realm","realm_epoch":3} 2021-02-11T00:36:05.588 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2021-02-11T00:36:08.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-02-11T00:37:05.589 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-42 2021-02-11T00:37:05.683 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-43 2021-02-11T00:37:05.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:37:05.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:37:05.733 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:37:05.771 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:37:05.771 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:37:05.973 INFO:teuthology.orchestra.run.smithi095.stderr:No --period given, using current period=4d4a6517-ce7a-42dd-9c62-65c684a638fb 2021-02-11T00:37:06.011 INFO:teuthology.orchestra.run.smithi095.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613003825.727943_198.1","last_update":"2021-02-11T00:37:05.727943Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:37:06.011 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2021-02-11T00:37:06.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:37:06.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:37:06.012 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:37:06.051 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:37:06.052 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:37:06.317 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"4d4a6517-ce7a-42dd-9c62-65c684a638fb","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":80,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"1_1613003825.727943_198.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:37:05.727943Z","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":80,"complete":80}} 2021-02-11T00:37:06.318 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2021-02-11T00:37:06.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:37:06.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:37:06.318 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:37:06.398 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:37:06.399 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:37:06.430 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:37:06.426+0000 7f72a0983080 0 NOTICE: overriding master zone: fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605 2021-02-11T00:37:06.431 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:37:06.426+0000 7f72a0983080 0 failed reading obj info from .rgw.root:zone_info.d88fea79-91d3-40e7-8bb7-515bb1e5c3b3: (2) No such file or directory 2021-02-11T00:37:06.431 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:37:06.426+0000 7f72a0983080 0 WARNING: could not read zone params for zone id=d88fea79-91d3-40e7-8bb7-515bb1e5c3b3 name=test-zone4 2021-02-11T00:37:06.433 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:37:06.430+0000 7f72a0983080 0 failed reading obj info from .rgw.root:zone_info.fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605: (2) No such file or directory 2021-02-11T00:37:06.433 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:37:06.430+0000 7f72a0983080 0 WARNING: could not read zone params for zone id=fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605 name=test-zone2 2021-02-11T00:37:06.447 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"CP2FO0ORQU6DSQKV","secret_key":"szUDueomrwAKvai4rUGz9pCzuDdncCWD"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","notif_pool":"test-zone1.rgw.log:notif"} 2021-02-11T00:37:06.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:37:06.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:37:06.448 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:37:06.527 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:37:06.527 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:37:06.848 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","epoch":1,"predecessor_uuid":"4d4a6517-ce7a-42dd-9c62-65c684a638fb","sync_status":["","1_1613003825.727943_198.1","",""],"period_map":{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","zones":[{"id":"35e954f9-5ea0-411e-8e41-e821d2787b31","name":"test-zone3","endpoints":["http://smithi057.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","name":"test-zone4","endpoints":["http://smithi095.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","endpoints":["http://smithi095.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"35e954f9-5ea0-411e-8e41-e821d2787b31","val":398828966},{"key":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","val":1248431312},{"key":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","val":3133685257},{"key":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","val":1241439546}]},"master_zonegroup":"test-zonegroup","master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","realm_name":"test-realm","realm_epoch":4} 2021-02-11T00:37:06.848 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2021-02-11T00:38:06.849 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-44 2021-02-11T00:38:06.890 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-45 2021-02-11T00:38:06.964 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-02-11T00:38:06.965 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-02-11T00:38:06.967 INFO:tasks.rgw.c1.client.1:Started 2021-02-11T00:38:06.967 DEBUG:teuthology.orchestra.run.smithi057:> curl http://smithi057.front.sepia.ceph.com:8001 2021-02-11T00:38:06.982 INFO:teuthology.orchestra.run.smithi057.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T00:38:06.983 INFO:teuthology.orchestra.run.smithi057.stderr: Dload Upload Total Spent Left Speed 2021-02-11T00:38:06.986 INFO:teuthology.orchestra.run.smithi057.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi057.front.sepia.ceph.com port 8001: Connection refused 2021-02-11T00:38:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-02-11T00:38:07.989 DEBUG:teuthology.orchestra.run.smithi057:> curl http://smithi057.front.sepia.ceph.com:8001 2021-02-11T00:38:08.003 INFO:teuthology.orchestra.run.smithi057.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-02-11T00:38:08.003 INFO:teuthology.orchestra.run.smithi057.stderr: Dload Upload Total Spent Left Speed 2021-02-11T00:38:08.009 INFO:teuthology.orchestra.run.smithi057.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 35666 2021-02-11T00:38:08.010 INFO:teuthology.orchestra.run.smithi057.stdout:anonymous 2021-02-11T00:38:08.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:08.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:08.011 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:08.094 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:08.095 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:08.278 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T00:38:08.318 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613003886.939234_199.1","last_update":"2021-02-11T00:38:06.939234Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:38:08.318 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:38:08.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:08.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:08.319 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:08.358 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:08.358 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:08.572 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003886.939234_199.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:06.939234Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:38:08.573 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:38:08.573 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:38:08.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:08.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:08.574 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:08.616 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:08.616 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:08.872 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003886.939234_199.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:06.939234Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:38:08.873 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:38:08.873 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:38:08.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:08.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:08.874 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:08.914 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:08.914 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:09.150 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003886.939234_199.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:06.939234Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:38:09.151 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:38:09.151 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-39 zones={test-zone2, test-zone1} 2021-02-11T00:38:09.172 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:09.223 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:09.285 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-39 zones={test-zone2, test-zone1} 2021-02-11T00:38:09.285 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-39 zones={test-zone3, test-zone1} 2021-02-11T00:38:09.302 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:09.350 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:09.414 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-39 zones={test-zone3, test-zone1} 2021-02-11T00:38:09.415 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-39 zones={test-zone1, test-zone2} 2021-02-11T00:38:09.425 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:09.471 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:09.533 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-39 zones={test-zone1, test-zone2} 2021-02-11T00:38:09.533 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-39 zones={test-zone3, test-zone2} 2021-02-11T00:38:09.543 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:09.590 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:09.654 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-39 zones={test-zone3, test-zone2} 2021-02-11T00:38:09.654 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-39 zones={test-zone1, test-zone3} 2021-02-11T00:38:09.667 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:09.715 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:10.758 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-39 zones={test-zone1, test-zone3} 2021-02-11T00:38:10.759 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-39 zones={test-zone2, test-zone3} 2021-02-11T00:38:10.769 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:10.814 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:10.879 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-39 zones={test-zone2, test-zone3} 2021-02-11T00:38:10.879 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-40 zones={test-zone2, test-zone1} 2021-02-11T00:38:10.893 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:10.942 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:11.005 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-40 zones={test-zone2, test-zone1} 2021-02-11T00:38:11.058 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-40 zones={test-zone3, test-zone1} 2021-02-11T00:38:11.074 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:11.122 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:11.182 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-40 zones={test-zone3, test-zone1} 2021-02-11T00:38:11.182 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-40 zones={test-zone1, test-zone2} 2021-02-11T00:38:11.193 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:11.238 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:11.302 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-40 zones={test-zone1, test-zone2} 2021-02-11T00:38:11.302 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-40 zones={test-zone3, test-zone2} 2021-02-11T00:38:11.312 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:11.359 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:11.421 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-40 zones={test-zone3, test-zone2} 2021-02-11T00:38:11.421 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-40 zones={test-zone1, test-zone3} 2021-02-11T00:38:11.433 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:11.478 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:11.543 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-40 zones={test-zone1, test-zone3} 2021-02-11T00:38:11.544 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-40 zones={test-zone2, test-zone3} 2021-02-11T00:38:11.553 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:11.603 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:11.690 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-40 zones={test-zone2, test-zone3} 2021-02-11T00:38:11.690 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-41 zones={test-zone2, test-zone1} 2021-02-11T00:38:11.706 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:11.754 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:11.816 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-41 zones={test-zone2, test-zone1} 2021-02-11T00:38:11.816 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-41 zones={test-zone3, test-zone1} 2021-02-11T00:38:11.829 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:11.874 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:11.931 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-41 zones={test-zone3, test-zone1} 2021-02-11T00:38:11.931 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-41 zones={test-zone1, test-zone2} 2021-02-11T00:38:11.940 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:11.986 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:12.050 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-41 zones={test-zone1, test-zone2} 2021-02-11T00:38:12.050 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-41 zones={test-zone3, test-zone2} 2021-02-11T00:38:12.060 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:12.110 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:12.173 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-41 zones={test-zone3, test-zone2} 2021-02-11T00:38:12.173 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-41 zones={test-zone1, test-zone3} 2021-02-11T00:38:12.184 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:12.234 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:12.303 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-41 zones={test-zone1, test-zone3} 2021-02-11T00:38:12.303 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-41 zones={test-zone2, test-zone3} 2021-02-11T00:38:12.319 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:12.367 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:12.430 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-41 zones={test-zone2, test-zone3} 2021-02-11T00:38:12.430 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-42 zones={test-zone2, test-zone1} 2021-02-11T00:38:12.439 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:12.487 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:12.549 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-42 zones={test-zone2, test-zone1} 2021-02-11T00:38:12.549 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-42 zones={test-zone3, test-zone1} 2021-02-11T00:38:12.559 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:12.606 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:12.664 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-42 zones={test-zone3, test-zone1} 2021-02-11T00:38:12.664 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-42 zones={test-zone1, test-zone2} 2021-02-11T00:38:12.675 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:12.723 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:12.786 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-42 zones={test-zone1, test-zone2} 2021-02-11T00:38:12.786 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-42 zones={test-zone3, test-zone2} 2021-02-11T00:38:12.796 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:12.847 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:12.910 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-42 zones={test-zone3, test-zone2} 2021-02-11T00:38:12.911 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-42 zones={test-zone1, test-zone3} 2021-02-11T00:38:12.918 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:12.963 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:13.026 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-42 zones={test-zone1, test-zone3} 2021-02-11T00:38:13.026 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-42 zones={test-zone2, test-zone3} 2021-02-11T00:38:13.039 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:13.087 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:13.144 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-42 zones={test-zone2, test-zone3} 2021-02-11T00:38:13.145 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-43 zones={test-zone2, test-zone1} 2021-02-11T00:38:13.153 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:13.203 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:13.267 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-43 zones={test-zone2, test-zone1} 2021-02-11T00:38:13.267 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-43 zones={test-zone3, test-zone1} 2021-02-11T00:38:13.276 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:13.323 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:13.381 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-43 zones={test-zone3, test-zone1} 2021-02-11T00:38:13.382 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-43 zones={test-zone1, test-zone2} 2021-02-11T00:38:13.393 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:13.449 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:13.535 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-43 zones={test-zone1, test-zone2} 2021-02-11T00:38:13.535 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-43 zones={test-zone3, test-zone2} 2021-02-11T00:38:13.569 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:13.651 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:13.753 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-43 zones={test-zone3, test-zone2} 2021-02-11T00:38:13.753 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-43 zones={test-zone1, test-zone3} 2021-02-11T00:38:13.778 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:13.827 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:13.918 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-43 zones={test-zone1, test-zone3} 2021-02-11T00:38:13.918 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-43 zones={test-zone2, test-zone3} 2021-02-11T00:38:13.930 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:13.979 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:14.050 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-43 zones={test-zone2, test-zone3} 2021-02-11T00:38:14.050 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-44 zones={test-zone2, test-zone1} 2021-02-11T00:38:14.061 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:14.107 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:14.170 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-44 zones={test-zone2, test-zone1} 2021-02-11T00:38:14.170 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-44 zones={test-zone3, test-zone1} 2021-02-11T00:38:14.179 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:14.227 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:14.284 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-44 zones={test-zone3, test-zone1} 2021-02-11T00:38:14.284 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-44 zones={test-zone1, test-zone2} 2021-02-11T00:38:14.295 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:14.342 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:14.410 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-44 zones={test-zone1, test-zone2} 2021-02-11T00:38:14.410 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-44 zones={test-zone3, test-zone2} 2021-02-11T00:38:14.420 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:14.467 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:14.529 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-44 zones={test-zone3, test-zone2} 2021-02-11T00:38:14.529 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-44 zones={test-zone1, test-zone3} 2021-02-11T00:38:14.541 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:14.587 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:14.650 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-44 zones={test-zone1, test-zone3} 2021-02-11T00:38:14.650 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-44 zones={test-zone2, test-zone3} 2021-02-11T00:38:14.659 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:14.707 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:14.773 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-44 zones={test-zone2, test-zone3} 2021-02-11T00:38:14.773 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-45 zones={test-zone2, test-zone1} 2021-02-11T00:38:14.783 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:14.830 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:14.894 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-45 zones={test-zone2, test-zone1} 2021-02-11T00:38:14.894 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-45 zones={test-zone3, test-zone1} 2021-02-11T00:38:14.904 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:14.950 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:15.013 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-45 zones={test-zone3, test-zone1} 2021-02-11T00:38:15.013 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-45 zones={test-zone1, test-zone2} 2021-02-11T00:38:15.024 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:15.070 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:15.133 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-45 zones={test-zone1, test-zone2} 2021-02-11T00:38:15.133 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-45 zones={test-zone3, test-zone2} 2021-02-11T00:38:15.141 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:15.190 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:15.253 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-45 zones={test-zone3, test-zone2} 2021-02-11T00:38:15.253 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-45 zones={test-zone1, test-zone3} 2021-02-11T00:38:15.266 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:15.315 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:15.377 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-45 zones={test-zone1, test-zone3} 2021-02-11T00:38:15.377 INFO:tasks.rgw_multi.zone_rados:comparing bucket=ufpbfn-45 zones={test-zone2, test-zone3} 2021-02-11T00:38:15.387 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-02-11T00:38:15.435 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-02-11T00:38:15.493 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=ufpbfn-45 zones={test-zone2, test-zone3} 2021-02-11T00:38:15.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '7f3547bf-73e3-43a7-aa91-648db0f982f6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:15.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '7f3547bf-73e3-43a7-aa91-648db0f982f6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:15.493 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 7f3547bf-73e3-43a7-aa91-648db0f982f6 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:15.531 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:15.531 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:15.802 INFO:teuthology.orchestra.run.smithi057.stdout:[{"id":"1_1613003450.854302_44.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T00:30:50.854302550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8afXW5L0598IonMpxK19QLU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003450.863150_45.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T00:30:50.863150983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8afXW5L0598IonMpxK19QLU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003573.921741_48.1","section":"bucket.instance","name":"ufpbfn-1:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.1","timestamp":"2021-02-11T00:32:53.921741821Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Skjph_QFacdJPxfoZZsFlwv","ver":1},"status":{"status":"write"}}},{"id":"1_1613003573.938336_49.1","section":"bucket.instance","name":"ufpbfn-1:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.1","timestamp":"2021-02-11T00:32:53.938336161Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Skjph_QFacdJPxfoZZsFlwv","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003573.939868_50.1","section":"bucket","name":"ufpbfn-1","timestamp":"2021-02-11T00:32:53.939868855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KqGKnzK6Tf8QvJEeDxt1_WH","ver":1},"status":{"status":"write"}}},{"id":"1_1613003573.944120_51.1","section":"bucket","name":"ufpbfn-1","timestamp":"2021-02-11T00:32:53.944120215Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KqGKnzK6Tf8QvJEeDxt1_WH","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003573.965387_52.1","section":"bucket.instance","name":"ufpbfn-2:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.2","timestamp":"2021-02-11T00:32:53.965387242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_opo5r9iAQSy3iT3OqymSbUR","ver":1},"status":{"status":"write"}}},{"id":"1_1613003573.970287_53.1","section":"bucket.instance","name":"ufpbfn-2:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.2","timestamp":"2021-02-11T00:32:53.970287398Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_opo5r9iAQSy3iT3OqymSbUR","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003573.971912_54.1","section":"bucket","name":"ufpbfn-2","timestamp":"2021-02-11T00:32:53.971912556Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I8Y2RjNUm2--MMKap47mPvi","ver":1},"status":{"status":"write"}}},{"id":"1_1613003573.976646_55.1","section":"bucket","name":"ufpbfn-2","timestamp":"2021-02-11T00:32:53.976646634Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I8Y2RjNUm2--MMKap47mPvi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003577.130440_56.1","section":"bucket.instance","name":"ufpbfn-3:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.3","timestamp":"2021-02-11T00:32:57.130440180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__JIYMfX43-rh93xo8qCfW3W","ver":1},"status":{"status":"write"}}},{"id":"1_1613003577.138907_57.1","section":"bucket.instance","name":"ufpbfn-3:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.3","timestamp":"2021-02-11T00:32:57.138907718Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__JIYMfX43-rh93xo8qCfW3W","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003577.141172_58.1","section":"bucket","name":"ufpbfn-3","timestamp":"2021-02-11T00:32:57.141172704Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZD69Kxj7MfN4ykEqn2AmMHP","ver":1},"status":{"status":"write"}}},{"id":"1_1613003577.148691_59.1","section":"bucket","name":"ufpbfn-3","timestamp":"2021-02-11T00:32:57.148691001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZD69Kxj7MfN4ykEqn2AmMHP","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.351288_60.1","section":"bucket.instance","name":"ufpbfn-4:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.4","timestamp":"2021-02-11T00:32:58.351288084Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7b_KGvalODv_9Uf9Kk5icn8","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.356570_61.1","section":"bucket.instance","name":"ufpbfn-4:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.4","timestamp":"2021-02-11T00:32:58.356570584Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7b_KGvalODv_9Uf9Kk5icn8","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.358632_62.1","section":"bucket","name":"ufpbfn-4","timestamp":"2021-02-11T00:32:58.358632530Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jr7oDJUzCTW6gHPoeufR7jm","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.363794_63.1","section":"bucket","name":"ufpbfn-4","timestamp":"2021-02-11T00:32:58.363794123Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jr7oDJUzCTW6gHPoeufR7jm","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.382871_64.1","section":"bucket.instance","name":"ufpbfn-5:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.5","timestamp":"2021-02-11T00:32:58.382871708Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a0Y62ZJ3YMY_wT-paa_QHFa","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.388437_65.1","section":"bucket.instance","name":"ufpbfn-5:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.5","timestamp":"2021-02-11T00:32:58.388437173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a0Y62ZJ3YMY_wT-paa_QHFa","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.390530_66.1","section":"bucket","name":"ufpbfn-5","timestamp":"2021-02-11T00:32:58.390530916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZMPDqfQwJlwt4fOPIl0M8aR","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.395610_67.1","section":"bucket","name":"ufpbfn-5","timestamp":"2021-02-11T00:32:58.395610305Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZMPDqfQwJlwt4fOPIl0M8aR","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.436947_68.1","section":"bucket.instance","name":"ufpbfn-6:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.6","timestamp":"2021-02-11T00:32:58.436947070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iVj6C98NL7g1XpDBkqjdwxs","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.442139_69.1","section":"bucket.instance","name":"ufpbfn-6:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.6","timestamp":"2021-02-11T00:32:58.442139964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iVj6C98NL7g1XpDBkqjdwxs","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.443881_70.1","section":"bucket","name":"ufpbfn-6","timestamp":"2021-02-11T00:32:58.443881886Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gm9dBc_bz3ThL3EzllaIRon","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.448531_71.1","section":"bucket","name":"ufpbfn-6","timestamp":"2021-02-11T00:32:58.448531161Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gm9dBc_bz3ThL3EzllaIRon","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.032175_72.1","section":"bucket.instance","name":"ufpbfn-7:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.7","timestamp":"2021-02-11T00:33:01.032175874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6WrCP2fZ4q43uXb3VOBaP-X","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.037898_73.1","section":"bucket.instance","name":"ufpbfn-7:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.7","timestamp":"2021-02-11T00:33:01.037898478Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6WrCP2fZ4q43uXb3VOBaP-X","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.040379_74.1","section":"bucket","name":"ufpbfn-7","timestamp":"2021-02-11T00:33:01.040379029Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.046480_75.1","section":"bucket","name":"ufpbfn-7","timestamp":"2021-02-11T00:33:01.046480658Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.066133_76.1","section":"bucket.instance","name":"ufpbfn-8:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.8","timestamp":"2021-02-11T00:33:01.066133971Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ImgVKlW5BV_4yXJ5eaN1blE","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.071786_77.1","section":"bucket.instance","name":"ufpbfn-8:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.8","timestamp":"2021-02-11T00:33:01.071786138Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ImgVKlW5BV_4yXJ5eaN1blE","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.073784_78.1","section":"bucket","name":"ufpbfn-8","timestamp":"2021-02-11T00:33:01.073784230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.078439_79.1","section":"bucket","name":"ufpbfn-8","timestamp":"2021-02-11T00:33:01.078439230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.118285_80.1","section":"bucket.instance","name":"ufpbfn-9:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.9","timestamp":"2021-02-11T00:33:01.118285107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4WCKMP2jWFDUs3dtJoGmuxJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.123404_81.1","section":"bucket.instance","name":"ufpbfn-9:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.9","timestamp":"2021-02-11T00:33:01.123404230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4WCKMP2jWFDUs3dtJoGmuxJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.125274_82.1","section":"bucket","name":"ufpbfn-9","timestamp":"2021-02-11T00:33:01.125274841Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.130865_83.1","section":"bucket","name":"ufpbfn-9","timestamp":"2021-02-11T00:33:01.130865699Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003582.332946_84.1","section":"bucket","name":"ufpbfn-7","timestamp":"2021-02-11T00:33:02.332946911Z","data":{"read_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":1},"write_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":2},"status":{"status":"remove"}}},{"id":"1_1613003582.340020_85.1","section":"bucket","name":"ufpbfn-7","timestamp":"2021-02-11T00:33:02.340020754Z","data":{"read_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":1},"write_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003582.420876_86.1","section":"bucket","name":"ufpbfn-8","timestamp":"2021-02-11T00:33:02.420876072Z","data":{"read_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":1},"write_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":2},"status":{"status":"remove"}}},{"id":"1_1613003582.427710_87.1","section":"bucket","name":"ufpbfn-8","timestamp":"2021-02-11T00:33:02.427710584Z","data":{"read_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":1},"write_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003582.554110_88.1","section":"bucket","name":"ufpbfn-9","timestamp":"2021-02-11T00:33:02.554110401Z","data":{"read_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":1},"write_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":2},"status":{"status":"remove"}}},{"id":"1_1613003582.559316_89.1","section":"bucket","name":"ufpbfn-9","timestamp":"2021-02-11T00:33:02.559316632Z","data":{"read_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":1},"write_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003583.730202_48.1","section":"bucket.instance","name":"ufpbfn-10:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.10","timestamp":"2021-02-11T00:33:03.730202256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2eGqXaG3AOEn7phUs2DcOIc","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.736098_58.1","section":"bucket.instance","name":"ufpbfn-10:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.10","timestamp":"2021-02-11T00:33:03.736098882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2eGqXaG3AOEn7phUs2DcOIc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.738359_59.1","section":"bucket","name":"ufpbfn-10","timestamp":"2021-02-11T00:33:03.738359979Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tDxlkqHxmS7PTDEab2H7hMT","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.744305_60.1","section":"bucket","name":"ufpbfn-10","timestamp":"2021-02-11T00:33:03.744305538Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tDxlkqHxmS7PTDEab2H7hMT","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.767444_61.1","section":"bucket.instance","name":"ufpbfn-11:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.11","timestamp":"2021-02-11T00:33:03.767444315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wkX6SG6rOg3K4s40k0F4qy5","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.773359_62.1","section":"bucket.instance","name":"ufpbfn-11:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.11","timestamp":"2021-02-11T00:33:03.773359088Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wkX6SG6rOg3K4s40k0F4qy5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.775379_63.1","section":"bucket","name":"ufpbfn-11","timestamp":"2021-02-11T00:33:03.775379051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U4YSB-MjK3Lwa5aeb8ZJsWC","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.780715_64.1","section":"bucket","name":"ufpbfn-11","timestamp":"2021-02-11T00:33:03.780715854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U4YSB-MjK3Lwa5aeb8ZJsWC","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.825241_65.1","section":"bucket.instance","name":"ufpbfn-12:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.12","timestamp":"2021-02-11T00:33:03.825241462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jq-H07HPoS-gctsMOEeF9RO","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.830032_66.1","section":"bucket.instance","name":"ufpbfn-12:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.12","timestamp":"2021-02-11T00:33:03.830032565Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jq-H07HPoS-gctsMOEeF9RO","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.832053_67.1","section":"bucket","name":"ufpbfn-12","timestamp":"2021-02-11T00:33:03.832053315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__fBfLcBSoKwIw5StU9Nvh6L","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.836924_68.1","section":"bucket","name":"ufpbfn-12","timestamp":"2021-02-11T00:33:03.836924032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__fBfLcBSoKwIw5StU9Nvh6L","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.659565_73.1","section":"bucket.instance","name":"ufpbfn-13:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.13","timestamp":"2021-02-11T00:33:17.659565432Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f8SIjk338xKgtCDgHkGKqff","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.665073_76.1","section":"bucket.instance","name":"ufpbfn-13:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.13","timestamp":"2021-02-11T00:33:17.665073710Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f8SIjk338xKgtCDgHkGKqff","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.666958_77.1","section":"bucket","name":"ufpbfn-13","timestamp":"2021-02-11T00:33:17.666958458Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0NQ3xjtUNQUmGx9jbXsmf2Z","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.671944_78.1","section":"bucket","name":"ufpbfn-13","timestamp":"2021-02-11T00:33:17.671944640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0NQ3xjtUNQUmGx9jbXsmf2Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.692682_79.1","section":"bucket.instance","name":"ufpbfn-14:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.14","timestamp":"2021-02-11T00:33:17.692682553Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L43RwNuj0ddF9ifaSHgIKye","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.699242_80.1","section":"bucket.instance","name":"ufpbfn-14:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.14","timestamp":"2021-02-11T00:33:17.699242273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L43RwNuj0ddF9ifaSHgIKye","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.701224_81.1","section":"bucket","name":"ufpbfn-14","timestamp":"2021-02-11T00:33:17.701224321Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MRG403-5B5xPKMkhFWKEsyk","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.706794_82.1","section":"bucket","name":"ufpbfn-14","timestamp":"2021-02-11T00:33:17.706794690Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MRG403-5B5xPKMkhFWKEsyk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.754649_83.1","section":"bucket.instance","name":"ufpbfn-15:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.15","timestamp":"2021-02-11T00:33:17.754649827Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KrcZ_amVznZV1n91M3anyap","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.765027_84.1","section":"bucket.instance","name":"ufpbfn-15:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.15","timestamp":"2021-02-11T00:33:17.765027652Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KrcZ_amVznZV1n91M3anyap","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.767014_85.1","section":"bucket","name":"ufpbfn-15","timestamp":"2021-02-11T00:33:17.767014291Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EgRyHS2CuJZW4_js4EagF7N","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.771457_86.1","section":"bucket","name":"ufpbfn-15","timestamp":"2021-02-11T00:33:17.771457040Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EgRyHS2CuJZW4_js4EagF7N","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.388153_87.1","section":"bucket.instance","name":"ufpbfn-16:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.16","timestamp":"2021-02-11T00:33:26.388153899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.394315_93.1","section":"bucket.instance","name":"ufpbfn-16:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.16","timestamp":"2021-02-11T00:33:26.394315586Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.396589_94.1","section":"bucket","name":"ufpbfn-16","timestamp":"2021-02-11T00:33:26.396589995Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c6TEuYH8lTtdHiiKB2PN7B3","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.401860_95.1","section":"bucket","name":"ufpbfn-16","timestamp":"2021-02-11T00:33:26.401860934Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c6TEuYH8lTtdHiiKB2PN7B3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.426046_96.1","section":"bucket.instance","name":"ufpbfn-17:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.17","timestamp":"2021-02-11T00:33:26.426046415Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.431552_97.1","section":"bucket.instance","name":"ufpbfn-17:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.17","timestamp":"2021-02-11T00:33:26.431552117Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.433354_98.1","section":"bucket","name":"ufpbfn-17","timestamp":"2021-02-11T00:33:26.433354162Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6elO0Q-rH28Qr3aVT2qAwx6","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.438176_99.1","section":"bucket","name":"ufpbfn-17","timestamp":"2021-02-11T00:33:26.438176837Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6elO0Q-rH28Qr3aVT2qAwx6","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.480439_101.1","section":"bucket.instance","name":"ufpbfn-18:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.18","timestamp":"2021-02-11T00:33:26.480439651Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.486235_102.1","section":"bucket.instance","name":"ufpbfn-18:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.18","timestamp":"2021-02-11T00:33:26.486235046Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.488060_103.1","section":"bucket","name":"ufpbfn-18","timestamp":"2021-02-11T00:33:26.488060035Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s4Ba4emrAs-SlpdvF9nVDgT","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.492951_104.1","section":"bucket","name":"ufpbfn-18","timestamp":"2021-02-11T00:33:26.492951271Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s4Ba4emrAs-SlpdvF9nVDgT","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.515036_105.1","section":"bucket.instance","name":"ufpbfn-16:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.16","timestamp":"2021-02-11T00:33:26.515036772Z","data":{"read_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":1},"write_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":2},"status":{"status":"write"}}},{"id":"1_1613003606.520310_106.1","section":"bucket.instance","name":"ufpbfn-16:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.16","timestamp":"2021-02-11T00:33:26.520310165Z","data":{"read_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":1},"write_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003606.525055_107.1","section":"bucket.instance","name":"ufpbfn-17:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.17","timestamp":"2021-02-11T00:33:26.525055588Z","data":{"read_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":1},"write_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":2},"status":{"status":"write"}}},{"id":"1_1613003606.531303_108.1","section":"bucket.instance","name":"ufpbfn-17:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.17","timestamp":"2021-02-11T00:33:26.531303087Z","data":{"read_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":1},"write_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003606.542749_110.1","section":"bucket.instance","name":"ufpbfn-18:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.18","timestamp":"2021-02-11T00:33:26.542749975Z","data":{"read_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":1},"write_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":2},"status":{"status":"write"}}},{"id":"1_1613003606.555218_111.1","section":"bucket.instance","name":"ufpbfn-18:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.18","timestamp":"2021-02-11T00:33:26.555218578Z","data":{"read_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":1},"write_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003663.038735_296.1","section":"bucket.instance","name":"ufpbfn-19:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.19","timestamp":"2021-02-11T00:34:23.038735977Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":1},"status":{"status":"write"}}},{"id":"1_1613003663.044326_299.1","section":"bucket.instance","name":"ufpbfn-19:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.19","timestamp":"2021-02-11T00:34:23.044326194Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003663.046352_300.1","section":"bucket","name":"ufpbfn-19","timestamp":"2021-02-11T00:34:23.046352692Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_quU3_y1npE6yjsgMfaiPzD-","ver":1},"status":{"status":"write"}}},{"id":"1_1613003663.051849_301.1","section":"bucket","name":"ufpbfn-19","timestamp":"2021-02-11T00:34:23.051849389Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_quU3_y1npE6yjsgMfaiPzD-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003663.058645_302.1","section":"bucket.instance","name":"ufpbfn-19:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.19","timestamp":"2021-02-11T00:34:23.058645404Z","data":{"read_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":1},"write_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":2},"status":{"status":"write"}}},{"id":"1_1613003663.066081_303.1","section":"bucket.instance","name":"ufpbfn-19:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.19","timestamp":"2021-02-11T00:34:23.066081205Z","data":{"read_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":1},"write_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003710.698748_304.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:10.698748192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":1},"status":{"status":"write"}}},{"id":"1_1613003710.704651_318.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:10.704651509Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003710.706704_319.1","section":"bucket","name":"ufpbfn-20","timestamp":"2021-02-11T00:35:10.706704347Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OoZxlXdV7j7UCHfz9fjr3nn","ver":1},"status":{"status":"write"}}},{"id":"1_1613003710.715573_320.1","section":"bucket","name":"ufpbfn-20","timestamp":"2021-02-11T00:35:10.715573590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OoZxlXdV7j7UCHfz9fjr3nn","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003719.828914_353.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:19.828914598Z","data":{"read_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":1},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":2},"status":{"status":"write"}}},{"id":"1_1613003719.838940_354.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:19.838940378Z","data":{"read_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":1},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003726.262512_399.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:26.262512788Z","data":{"read_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":2},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":3},"status":{"status":"write"}}},{"id":"1_1613003726.272815_400.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:26.272815115Z","data":{"read_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":2},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":3},"status":{"status":"complete"}}},{"id":"1_1613003730.762664_445.1","section":"bucket.instance","name":"ufpbfn-21:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.21","timestamp":"2021-02-11T00:35:30.762664019Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.772882_446.1","section":"bucket.instance","name":"ufpbfn-21:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.21","timestamp":"2021-02-11T00:35:30.772882870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.775134_447.1","section":"bucket","name":"ufpbfn-21","timestamp":"2021-02-11T00:35:30.775134980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k5uY3Gk97MZZN-lG6Hb8Fye","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.783979_448.1","section":"bucket","name":"ufpbfn-21","timestamp":"2021-02-11T00:35:30.783979099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k5uY3Gk97MZZN-lG6Hb8Fye","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.811133_449.1","section":"bucket.instance","name":"ufpbfn-22:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.22","timestamp":"2021-02-11T00:35:30.811133053Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.816622_450.1","section":"bucket.instance","name":"ufpbfn-22:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.22","timestamp":"2021-02-11T00:35:30.816622426Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.818531_451.1","section":"bucket","name":"ufpbfn-22","timestamp":"2021-02-11T00:35:30.818531149Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FC7QfdFkMCENzYwmhnYnLvS","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.823109_452.1","section":"bucket","name":"ufpbfn-22","timestamp":"2021-02-11T00:35:30.823109099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FC7QfdFkMCENzYwmhnYnLvS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.866148_453.1","section":"bucket.instance","name":"ufpbfn-23:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.23","timestamp":"2021-02-11T00:35:30.866148114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.871906_454.1","section":"bucket.instance","name":"ufpbfn-23:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.23","timestamp":"2021-02-11T00:35:30.871906109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.875362_455.1","section":"bucket","name":"ufpbfn-23","timestamp":"2021-02-11T00:35:30.875362846Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JXU6nBLcF_sOtZUty9mXONE","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.880899_456.1","section":"bucket","name":"ufpbfn-23","timestamp":"2021-02-11T00:35:30.880899190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JXU6nBLcF_sOtZUty9mXONE","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.906538_457.1","section":"bucket.instance","name":"ufpbfn-21:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.21","timestamp":"2021-02-11T00:35:30.906538677Z","data":{"read_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":1},"write_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":2},"status":{"status":"write"}}},{"id":"1_1613003730.912385_458.1","section":"bucket.instance","name":"ufpbfn-21:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.21","timestamp":"2021-02-11T00:35:30.912385378Z","data":{"read_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":1},"write_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003730.916595_459.1","section":"bucket.instance","name":"ufpbfn-22:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.22","timestamp":"2021-02-11T00:35:30.916595556Z","data":{"read_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":1},"write_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":2},"status":{"status":"write"}}},{"id":"1_1613003730.921350_460.1","section":"bucket.instance","name":"ufpbfn-22:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.22","timestamp":"2021-02-11T00:35:30.921350521Z","data":{"read_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":1},"write_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003730.930587_461.1","section":"bucket.instance","name":"ufpbfn-23:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.23","timestamp":"2021-02-11T00:35:30.930587206Z","data":{"read_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":1},"write_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":2},"status":{"status":"write"}}},{"id":"1_1613003730.936764_462.1","section":"bucket.instance","name":"ufpbfn-23:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.23","timestamp":"2021-02-11T00:35:30.936764259Z","data":{"read_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":1},"write_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003742.361724_463.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.361724903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.367316_471.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.367316098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.369891_472.1","section":"bucket","name":"ufpbfn-24","timestamp":"2021-02-11T00:35:42.369891133Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3K3JZc4KLLW9-gE3essIbqt","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.375942_473.1","section":"bucket","name":"ufpbfn-24","timestamp":"2021-02-11T00:35:42.375942599Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3K3JZc4KLLW9-gE3essIbqt","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.404482_474.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.404482219Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.410336_475.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.410336039Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.412835_476.1","section":"bucket","name":"ufpbfn-25","timestamp":"2021-02-11T00:35:42.412835404Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a-6lD0m5KM_3p3bHNRp9pQd","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.417918_477.1","section":"bucket","name":"ufpbfn-25","timestamp":"2021-02-11T00:35:42.417918083Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a-6lD0m5KM_3p3bHNRp9pQd","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.451495_478.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.451495596Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.458335_479.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.458335827Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.463725_480.1","section":"bucket","name":"ufpbfn-26","timestamp":"2021-02-11T00:35:42.463725770Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_weJpVYgvsyih4Mc-CA_rT6D","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.471367_481.1","section":"bucket","name":"ufpbfn-26","timestamp":"2021-02-11T00:35:42.471367127Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_weJpVYgvsyih4Mc-CA_rT6D","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.492942_482.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.492942914Z","data":{"read_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":1},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":2},"status":{"status":"write"}}},{"id":"1_1613003742.498602_483.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.498602849Z","data":{"read_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":1},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003742.502980_484.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.502980249Z","data":{"read_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":2},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":3},"status":{"status":"write"}}},{"id":"1_1613003742.509352_485.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.509352182Z","data":{"read_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":2},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":3},"status":{"status":"complete"}}},{"id":"1_1613003742.515116_486.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.515116187Z","data":{"read_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":1},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":2},"status":{"status":"write"}}},{"id":"1_1613003742.522854_487.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.522854749Z","data":{"read_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":1},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003742.535194_488.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.535194409Z","data":{"read_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":2},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":3},"status":{"status":"write"}}},{"id":"1_1613003742.544167_489.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.544167180Z","data":{"read_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":2},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":3},"status":{"status":"complete"}}},{"id":"1_1613003742.556349_490.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.556349382Z","data":{"read_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":1},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":2},"status":{"status":"write"}}},{"id":"1_1613003742.566275_491.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.566275024Z","data":{"read_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":1},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003742.578851_492.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.578851542Z","data":{"read_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":2},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":3},"status":{"status":"write"}}},{"id":"1_1613003742.584739_493.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.584739159Z","data":{"read_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":2},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":3},"status":{"status":"complete"}}},{"id":"1_1613003752.114576_494.1","section":"bucket.instance","name":"ufpbfn-27:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.27","timestamp":"2021-02-11T00:35:52.114576760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.120282_502.1","section":"bucket.instance","name":"ufpbfn-27:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.27","timestamp":"2021-02-11T00:35:52.120282248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.122353_503.1","section":"bucket","name":"ufpbfn-27","timestamp":"2021-02-11T00:35:52.122353726Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wtfPlNxQltZmQNvUExRJKda","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.127478_504.1","section":"bucket","name":"ufpbfn-27","timestamp":"2021-02-11T00:35:52.127478495Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wtfPlNxQltZmQNvUExRJKda","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.150327_505.1","section":"bucket.instance","name":"ufpbfn-28:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.28","timestamp":"2021-02-11T00:35:52.150327964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.156621_506.1","section":"bucket.instance","name":"ufpbfn-28:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.28","timestamp":"2021-02-11T00:35:52.156621989Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.158759_507.1","section":"bucket","name":"ufpbfn-28","timestamp":"2021-02-11T00:35:52.158759219Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yn9gTjGHQR-RBwC8BZeqo8y","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.164363_508.1","section":"bucket","name":"ufpbfn-28","timestamp":"2021-02-11T00:35:52.164363564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yn9gTjGHQR-RBwC8BZeqo8y","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.210610_509.1","section":"bucket.instance","name":"ufpbfn-29:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.29","timestamp":"2021-02-11T00:35:52.210610079Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.218162_510.1","section":"bucket.instance","name":"ufpbfn-29:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.29","timestamp":"2021-02-11T00:35:52.218162393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.221895_511.1","section":"bucket","name":"ufpbfn-29","timestamp":"2021-02-11T00:35:52.221895004Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JVPKNH1TNGfzqm12utWovVB","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.232622_512.1","section":"bucket","name":"ufpbfn-29","timestamp":"2021-02-11T00:35:52.232622253Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JVPKNH1TNGfzqm12utWovVB","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.255266_513.1","section":"bucket.instance","name":"ufpbfn-27:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.27","timestamp":"2021-02-11T00:35:52.255266759Z","data":{"read_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":1},"write_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.265160_514.1","section":"bucket.instance","name":"ufpbfn-27:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.27","timestamp":"2021-02-11T00:35:52.265160032Z","data":{"read_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":1},"write_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.315161_515.1","section":"bucket.instance","name":"ufpbfn-28:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.28","timestamp":"2021-02-11T00:35:52.315161773Z","data":{"read_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":1},"write_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.325134_516.1","section":"bucket.instance","name":"ufpbfn-28:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.28","timestamp":"2021-02-11T00:35:52.325134952Z","data":{"read_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":1},"write_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.382688_517.1","section":"bucket.instance","name":"ufpbfn-29:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.29","timestamp":"2021-02-11T00:35:52.382688910Z","data":{"read_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":1},"write_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.393754_518.1","section":"bucket.instance","name":"ufpbfn-29:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.29","timestamp":"2021-02-11T00:35:52.393754381Z","data":{"read_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":1},"write_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.469936_519.1","section":"bucket.instance","name":"ufpbfn-30:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.30","timestamp":"2021-02-11T00:35:52.469936591Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.475466_520.1","section":"bucket.instance","name":"ufpbfn-30:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.30","timestamp":"2021-02-11T00:35:52.475466058Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.477315_521.1","section":"bucket","name":"ufpbfn-30","timestamp":"2021-02-11T00:35:52.477315559Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jjRH-goef-lAu1Abk5YXeix","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.481744_522.1","section":"bucket","name":"ufpbfn-30","timestamp":"2021-02-11T00:35:52.481744078Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jjRH-goef-lAu1Abk5YXeix","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.502453_523.1","section":"bucket.instance","name":"ufpbfn-31:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.31","timestamp":"2021-02-11T00:35:52.502453639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.508136_524.1","section":"bucket.instance","name":"ufpbfn-31:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.31","timestamp":"2021-02-11T00:35:52.508136124Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.510150_525.1","section":"bucket","name":"ufpbfn-31","timestamp":"2021-02-11T00:35:52.510150885Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YkTpeSeHlvxSYw5Rw8Ab4_g","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.515456_526.1","section":"bucket","name":"ufpbfn-31","timestamp":"2021-02-11T00:35:52.515456566Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YkTpeSeHlvxSYw5Rw8Ab4_g","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.561311_527.1","section":"bucket.instance","name":"ufpbfn-32:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.32","timestamp":"2021-02-11T00:35:52.561311410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.567433_528.1","section":"bucket.instance","name":"ufpbfn-32:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.32","timestamp":"2021-02-11T00:35:52.567433316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.569487_529.1","section":"bucket","name":"ufpbfn-32","timestamp":"2021-02-11T00:35:52.569487932Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LHW-hTrz97v-1TYWLUyydCX","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.574147_530.1","section":"bucket","name":"ufpbfn-32","timestamp":"2021-02-11T00:35:52.574147158Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LHW-hTrz97v-1TYWLUyydCX","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.597912_531.1","section":"bucket.instance","name":"ufpbfn-30:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.30","timestamp":"2021-02-11T00:35:52.597912475Z","data":{"read_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":1},"write_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.604469_532.1","section":"bucket.instance","name":"ufpbfn-30:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.30","timestamp":"2021-02-11T00:35:52.604469492Z","data":{"read_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":1},"write_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.656328_533.1","section":"bucket.instance","name":"ufpbfn-31:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.31","timestamp":"2021-02-11T00:35:52.656328101Z","data":{"read_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":1},"write_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.668788_534.1","section":"bucket.instance","name":"ufpbfn-31:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.31","timestamp":"2021-02-11T00:35:52.668788500Z","data":{"read_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":1},"write_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.724450_535.1","section":"bucket.instance","name":"ufpbfn-32:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.32","timestamp":"2021-02-11T00:35:52.724450815Z","data":{"read_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":1},"write_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.732591_536.1","section":"bucket.instance","name":"ufpbfn-32:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.32","timestamp":"2021-02-11T00:35:52.732591666Z","data":{"read_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":1},"write_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.808374_537.1","section":"bucket.instance","name":"ufpbfn-33:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.33","timestamp":"2021-02-11T00:35:52.808374402Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.815859_538.1","section":"bucket.instance","name":"ufpbfn-33:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.33","timestamp":"2021-02-11T00:35:52.815859569Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.821293_539.1","section":"bucket","name":"ufpbfn-33","timestamp":"2021-02-11T00:35:52.821293819Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qyZG5IHPnJ0q7XL8oHBFG_M","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.833861_540.1","section":"bucket","name":"ufpbfn-33","timestamp":"2021-02-11T00:35:52.833861919Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qyZG5IHPnJ0q7XL8oHBFG_M","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.855079_541.1","section":"bucket.instance","name":"ufpbfn-34:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.34","timestamp":"2021-02-11T00:35:52.855079273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.863679_542.1","section":"bucket.instance","name":"ufpbfn-34:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.34","timestamp":"2021-02-11T00:35:52.863679302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.865715_543.1","section":"bucket","name":"ufpbfn-34","timestamp":"2021-02-11T00:35:52.865715392Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eaPmsUe9UpKZde4q81ZEGn-","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.870203_544.1","section":"bucket","name":"ufpbfn-34","timestamp":"2021-02-11T00:35:52.870203091Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eaPmsUe9UpKZde4q81ZEGn-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.911282_545.1","section":"bucket.instance","name":"ufpbfn-35:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.35","timestamp":"2021-02-11T00:35:52.911282623Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.917850_546.1","section":"bucket.instance","name":"ufpbfn-35:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.35","timestamp":"2021-02-11T00:35:52.917850358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.920196_547.1","section":"bucket","name":"ufpbfn-35","timestamp":"2021-02-11T00:35:52.920196626Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2P7hUA6fxJFof13fab6ELfJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.925674_548.1","section":"bucket","name":"ufpbfn-35","timestamp":"2021-02-11T00:35:52.925674365Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2P7hUA6fxJFof13fab6ELfJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.946987_549.1","section":"bucket.instance","name":"ufpbfn-33:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.33","timestamp":"2021-02-11T00:35:52.946987783Z","data":{"read_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":1},"write_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.952944_550.1","section":"bucket.instance","name":"ufpbfn-33:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.33","timestamp":"2021-02-11T00:35:52.952944552Z","data":{"read_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":1},"write_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003754.007637_551.1","section":"bucket.instance","name":"ufpbfn-34:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.34","timestamp":"2021-02-11T00:35:54.007637789Z","data":{"read_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":1},"write_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":2},"status":{"status":"write"}}},{"id":"1_1613003754.017772_552.1","section":"bucket.instance","name":"ufpbfn-34:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.34","timestamp":"2021-02-11T00:35:54.017772278Z","data":{"read_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":1},"write_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003755.079943_553.1","section":"bucket.instance","name":"ufpbfn-35:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.35","timestamp":"2021-02-11T00:35:55.079943426Z","data":{"read_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":1},"write_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":2},"status":{"status":"write"}}},{"id":"1_1613003755.092005_554.1","section":"bucket.instance","name":"ufpbfn-35:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.35","timestamp":"2021-02-11T00:35:55.092005034Z","data":{"read_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":1},"write_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003755.170194_555.1","section":"bucket.instance","name":"ufpbfn-36:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.36","timestamp":"2021-02-11T00:35:55.170194935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6Bhkdo_dUOAdMeAbwoirslC","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.177626_556.1","section":"bucket.instance","name":"ufpbfn-36:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.36","timestamp":"2021-02-11T00:35:55.177626644Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6Bhkdo_dUOAdMeAbwoirslC","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.180328_557.1","section":"bucket","name":"ufpbfn-36","timestamp":"2021-02-11T00:35:55.180328782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yhkND0d6ef1KagNiHqzkC3_","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.186832_558.1","section":"bucket","name":"ufpbfn-36","timestamp":"2021-02-11T00:35:55.186832130Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yhkND0d6ef1KagNiHqzkC3_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.224261_559.1","section":"bucket.instance","name":"ufpbfn-37:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.37","timestamp":"2021-02-11T00:35:55.224261590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j2fBBQD3zN4CmLC7L14TrCt","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.232953_560.1","section":"bucket.instance","name":"ufpbfn-37:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.37","timestamp":"2021-02-11T00:35:55.232953898Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j2fBBQD3zN4CmLC7L14TrCt","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.235184_561.1","section":"bucket","name":"ufpbfn-37","timestamp":"2021-02-11T00:35:55.235184111Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ET2FdSdbJ8ZEOCAPZJ-po8a","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.239904_562.1","section":"bucket","name":"ufpbfn-37","timestamp":"2021-02-11T00:35:55.239904921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ET2FdSdbJ8ZEOCAPZJ-po8a","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.279481_563.1","section":"bucket.instance","name":"ufpbfn-38:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.38","timestamp":"2021-02-11T00:35:55.279481119Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DmS3IetBwqoZFY34Xk3Ohek","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.284473_564.1","section":"bucket.instance","name":"ufpbfn-38:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.38","timestamp":"2021-02-11T00:35:55.284473066Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DmS3IetBwqoZFY34Xk3Ohek","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.286540_565.1","section":"bucket","name":"ufpbfn-38","timestamp":"2021-02-11T00:35:55.286540135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0ru7BE0bzAXx0-Pb22dsKmU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.291866_566.1","section":"bucket","name":"ufpbfn-38","timestamp":"2021-02-11T00:35:55.291866653Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0ru7BE0bzAXx0-Pb22dsKmU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.788416_570.1","section":"bucket.instance","name":"ufpbfn-39:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.39","timestamp":"2021-02-11T00:35:57.788416591Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1vgWQA2CxVJU7vO6qXuqW2w","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.796322_571.1","section":"bucket.instance","name":"ufpbfn-39:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.39","timestamp":"2021-02-11T00:35:57.796322629Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1vgWQA2CxVJU7vO6qXuqW2w","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.798608_572.1","section":"bucket","name":"ufpbfn-39","timestamp":"2021-02-11T00:35:57.798608517Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kSxSKKo8ruuBEqht7yFAiKh","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.804310_573.1","section":"bucket","name":"ufpbfn-39","timestamp":"2021-02-11T00:35:57.804310795Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kSxSKKo8ruuBEqht7yFAiKh","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.834923_574.1","section":"bucket.instance","name":"ufpbfn-40:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.40","timestamp":"2021-02-11T00:35:57.834923842Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jcrlr-QMPmHhG1khCqUK-HL","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.844273_575.1","section":"bucket.instance","name":"ufpbfn-40:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.40","timestamp":"2021-02-11T00:35:57.844273207Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jcrlr-QMPmHhG1khCqUK-HL","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.846725_576.1","section":"bucket","name":"ufpbfn-40","timestamp":"2021-02-11T00:35:57.846725006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-U2oP92YIGzVDnZTkauQDfU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.851372_577.1","section":"bucket","name":"ufpbfn-40","timestamp":"2021-02-11T00:35:57.851372809Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-U2oP92YIGzVDnZTkauQDfU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.893851_578.1","section":"bucket.instance","name":"ufpbfn-41:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.41","timestamp":"2021-02-11T00:35:57.893851494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_F6KHu-saDEKBY8ttlAXOWyy","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.902224_579.1","section":"bucket.instance","name":"ufpbfn-41:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.41","timestamp":"2021-02-11T00:35:57.902224837Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_F6KHu-saDEKBY8ttlAXOWyy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.904331_580.1","section":"bucket","name":"ufpbfn-41","timestamp":"2021-02-11T00:35:57.904331122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Hr9JbpKcJAgiTFrG2I388EO","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.913330_581.1","section":"bucket","name":"ufpbfn-41","timestamp":"2021-02-11T00:35:57.913330476Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Hr9JbpKcJAgiTFrG2I388EO","ver":1},"status":{"status":"complete"}}}] 2021-02-11T00:38:15.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '7f3547bf-73e3-43a7-aa91-648db0f982f6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:15.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '7f3547bf-73e3-43a7-aa91-648db0f982f6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:15.804 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 7f3547bf-73e3-43a7-aa91-648db0f982f6 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:15.845 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:15.845 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:16.089 INFO:teuthology.orchestra.run.smithi095.stdout:[{"id":"1_1613003450.854302_44.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T00:30:50.854302550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8afXW5L0598IonMpxK19QLU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003450.863150_45.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T00:30:50.863150983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8afXW5L0598IonMpxK19QLU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003573.921741_48.1","section":"bucket.instance","name":"ufpbfn-1:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.1","timestamp":"2021-02-11T00:32:53.921741821Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Skjph_QFacdJPxfoZZsFlwv","ver":1},"status":{"status":"write"}}},{"id":"1_1613003573.938336_49.1","section":"bucket.instance","name":"ufpbfn-1:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.1","timestamp":"2021-02-11T00:32:53.938336161Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Skjph_QFacdJPxfoZZsFlwv","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003573.939868_50.1","section":"bucket","name":"ufpbfn-1","timestamp":"2021-02-11T00:32:53.939868855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KqGKnzK6Tf8QvJEeDxt1_WH","ver":1},"status":{"status":"write"}}},{"id":"1_1613003573.944120_51.1","section":"bucket","name":"ufpbfn-1","timestamp":"2021-02-11T00:32:53.944120215Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KqGKnzK6Tf8QvJEeDxt1_WH","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003573.965387_52.1","section":"bucket.instance","name":"ufpbfn-2:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.2","timestamp":"2021-02-11T00:32:53.965387242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_opo5r9iAQSy3iT3OqymSbUR","ver":1},"status":{"status":"write"}}},{"id":"1_1613003573.970287_53.1","section":"bucket.instance","name":"ufpbfn-2:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.2","timestamp":"2021-02-11T00:32:53.970287398Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_opo5r9iAQSy3iT3OqymSbUR","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003573.971912_54.1","section":"bucket","name":"ufpbfn-2","timestamp":"2021-02-11T00:32:53.971912556Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I8Y2RjNUm2--MMKap47mPvi","ver":1},"status":{"status":"write"}}},{"id":"1_1613003573.976646_55.1","section":"bucket","name":"ufpbfn-2","timestamp":"2021-02-11T00:32:53.976646634Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I8Y2RjNUm2--MMKap47mPvi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003577.130440_56.1","section":"bucket.instance","name":"ufpbfn-3:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.3","timestamp":"2021-02-11T00:32:57.130440180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__JIYMfX43-rh93xo8qCfW3W","ver":1},"status":{"status":"write"}}},{"id":"1_1613003577.138907_57.1","section":"bucket.instance","name":"ufpbfn-3:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.3","timestamp":"2021-02-11T00:32:57.138907718Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__JIYMfX43-rh93xo8qCfW3W","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003577.141172_58.1","section":"bucket","name":"ufpbfn-3","timestamp":"2021-02-11T00:32:57.141172704Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZD69Kxj7MfN4ykEqn2AmMHP","ver":1},"status":{"status":"write"}}},{"id":"1_1613003577.148691_59.1","section":"bucket","name":"ufpbfn-3","timestamp":"2021-02-11T00:32:57.148691001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZD69Kxj7MfN4ykEqn2AmMHP","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.351288_60.1","section":"bucket.instance","name":"ufpbfn-4:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.4","timestamp":"2021-02-11T00:32:58.351288084Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7b_KGvalODv_9Uf9Kk5icn8","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.356570_61.1","section":"bucket.instance","name":"ufpbfn-4:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.4","timestamp":"2021-02-11T00:32:58.356570584Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7b_KGvalODv_9Uf9Kk5icn8","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.358632_62.1","section":"bucket","name":"ufpbfn-4","timestamp":"2021-02-11T00:32:58.358632530Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jr7oDJUzCTW6gHPoeufR7jm","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.363794_63.1","section":"bucket","name":"ufpbfn-4","timestamp":"2021-02-11T00:32:58.363794123Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jr7oDJUzCTW6gHPoeufR7jm","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.382871_64.1","section":"bucket.instance","name":"ufpbfn-5:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.5","timestamp":"2021-02-11T00:32:58.382871708Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a0Y62ZJ3YMY_wT-paa_QHFa","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.388437_65.1","section":"bucket.instance","name":"ufpbfn-5:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.5","timestamp":"2021-02-11T00:32:58.388437173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a0Y62ZJ3YMY_wT-paa_QHFa","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.390530_66.1","section":"bucket","name":"ufpbfn-5","timestamp":"2021-02-11T00:32:58.390530916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZMPDqfQwJlwt4fOPIl0M8aR","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.395610_67.1","section":"bucket","name":"ufpbfn-5","timestamp":"2021-02-11T00:32:58.395610305Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZMPDqfQwJlwt4fOPIl0M8aR","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.436947_68.1","section":"bucket.instance","name":"ufpbfn-6:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.6","timestamp":"2021-02-11T00:32:58.436947070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iVj6C98NL7g1XpDBkqjdwxs","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.442139_69.1","section":"bucket.instance","name":"ufpbfn-6:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.6","timestamp":"2021-02-11T00:32:58.442139964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iVj6C98NL7g1XpDBkqjdwxs","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.443881_70.1","section":"bucket","name":"ufpbfn-6","timestamp":"2021-02-11T00:32:58.443881886Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gm9dBc_bz3ThL3EzllaIRon","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.448531_71.1","section":"bucket","name":"ufpbfn-6","timestamp":"2021-02-11T00:32:58.448531161Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gm9dBc_bz3ThL3EzllaIRon","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.032175_72.1","section":"bucket.instance","name":"ufpbfn-7:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.7","timestamp":"2021-02-11T00:33:01.032175874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6WrCP2fZ4q43uXb3VOBaP-X","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.037898_73.1","section":"bucket.instance","name":"ufpbfn-7:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.7","timestamp":"2021-02-11T00:33:01.037898478Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6WrCP2fZ4q43uXb3VOBaP-X","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.040379_74.1","section":"bucket","name":"ufpbfn-7","timestamp":"2021-02-11T00:33:01.040379029Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.046480_75.1","section":"bucket","name":"ufpbfn-7","timestamp":"2021-02-11T00:33:01.046480658Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.066133_76.1","section":"bucket.instance","name":"ufpbfn-8:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.8","timestamp":"2021-02-11T00:33:01.066133971Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ImgVKlW5BV_4yXJ5eaN1blE","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.071786_77.1","section":"bucket.instance","name":"ufpbfn-8:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.8","timestamp":"2021-02-11T00:33:01.071786138Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ImgVKlW5BV_4yXJ5eaN1blE","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.073784_78.1","section":"bucket","name":"ufpbfn-8","timestamp":"2021-02-11T00:33:01.073784230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.078439_79.1","section":"bucket","name":"ufpbfn-8","timestamp":"2021-02-11T00:33:01.078439230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.118285_80.1","section":"bucket.instance","name":"ufpbfn-9:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.9","timestamp":"2021-02-11T00:33:01.118285107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4WCKMP2jWFDUs3dtJoGmuxJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.123404_81.1","section":"bucket.instance","name":"ufpbfn-9:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.9","timestamp":"2021-02-11T00:33:01.123404230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4WCKMP2jWFDUs3dtJoGmuxJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.125274_82.1","section":"bucket","name":"ufpbfn-9","timestamp":"2021-02-11T00:33:01.125274841Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.130865_83.1","section":"bucket","name":"ufpbfn-9","timestamp":"2021-02-11T00:33:01.130865699Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003582.332946_84.1","section":"bucket","name":"ufpbfn-7","timestamp":"2021-02-11T00:33:02.332946911Z","data":{"read_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":1},"write_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":2},"status":{"status":"remove"}}},{"id":"1_1613003582.340020_85.1","section":"bucket","name":"ufpbfn-7","timestamp":"2021-02-11T00:33:02.340020754Z","data":{"read_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":1},"write_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003582.420876_86.1","section":"bucket","name":"ufpbfn-8","timestamp":"2021-02-11T00:33:02.420876072Z","data":{"read_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":1},"write_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":2},"status":{"status":"remove"}}},{"id":"1_1613003582.427710_87.1","section":"bucket","name":"ufpbfn-8","timestamp":"2021-02-11T00:33:02.427710584Z","data":{"read_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":1},"write_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003582.554110_88.1","section":"bucket","name":"ufpbfn-9","timestamp":"2021-02-11T00:33:02.554110401Z","data":{"read_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":1},"write_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":2},"status":{"status":"remove"}}},{"id":"1_1613003582.559316_89.1","section":"bucket","name":"ufpbfn-9","timestamp":"2021-02-11T00:33:02.559316632Z","data":{"read_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":1},"write_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003583.730202_48.1","section":"bucket.instance","name":"ufpbfn-10:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.10","timestamp":"2021-02-11T00:33:03.730202256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2eGqXaG3AOEn7phUs2DcOIc","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.736098_58.1","section":"bucket.instance","name":"ufpbfn-10:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.10","timestamp":"2021-02-11T00:33:03.736098882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2eGqXaG3AOEn7phUs2DcOIc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.738359_59.1","section":"bucket","name":"ufpbfn-10","timestamp":"2021-02-11T00:33:03.738359979Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tDxlkqHxmS7PTDEab2H7hMT","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.744305_60.1","section":"bucket","name":"ufpbfn-10","timestamp":"2021-02-11T00:33:03.744305538Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tDxlkqHxmS7PTDEab2H7hMT","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.767444_61.1","section":"bucket.instance","name":"ufpbfn-11:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.11","timestamp":"2021-02-11T00:33:03.767444315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wkX6SG6rOg3K4s40k0F4qy5","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.773359_62.1","section":"bucket.instance","name":"ufpbfn-11:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.11","timestamp":"2021-02-11T00:33:03.773359088Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wkX6SG6rOg3K4s40k0F4qy5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.775379_63.1","section":"bucket","name":"ufpbfn-11","timestamp":"2021-02-11T00:33:03.775379051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U4YSB-MjK3Lwa5aeb8ZJsWC","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.780715_64.1","section":"bucket","name":"ufpbfn-11","timestamp":"2021-02-11T00:33:03.780715854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U4YSB-MjK3Lwa5aeb8ZJsWC","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.825241_65.1","section":"bucket.instance","name":"ufpbfn-12:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.12","timestamp":"2021-02-11T00:33:03.825241462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jq-H07HPoS-gctsMOEeF9RO","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.830032_66.1","section":"bucket.instance","name":"ufpbfn-12:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.12","timestamp":"2021-02-11T00:33:03.830032565Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jq-H07HPoS-gctsMOEeF9RO","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.832053_67.1","section":"bucket","name":"ufpbfn-12","timestamp":"2021-02-11T00:33:03.832053315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__fBfLcBSoKwIw5StU9Nvh6L","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.836924_68.1","section":"bucket","name":"ufpbfn-12","timestamp":"2021-02-11T00:33:03.836924032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__fBfLcBSoKwIw5StU9Nvh6L","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.659565_73.1","section":"bucket.instance","name":"ufpbfn-13:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.13","timestamp":"2021-02-11T00:33:17.659565432Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f8SIjk338xKgtCDgHkGKqff","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.665073_76.1","section":"bucket.instance","name":"ufpbfn-13:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.13","timestamp":"2021-02-11T00:33:17.665073710Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f8SIjk338xKgtCDgHkGKqff","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.666958_77.1","section":"bucket","name":"ufpbfn-13","timestamp":"2021-02-11T00:33:17.666958458Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0NQ3xjtUNQUmGx9jbXsmf2Z","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.671944_78.1","section":"bucket","name":"ufpbfn-13","timestamp":"2021-02-11T00:33:17.671944640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0NQ3xjtUNQUmGx9jbXsmf2Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.692682_79.1","section":"bucket.instance","name":"ufpbfn-14:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.14","timestamp":"2021-02-11T00:33:17.692682553Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L43RwNuj0ddF9ifaSHgIKye","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.699242_80.1","section":"bucket.instance","name":"ufpbfn-14:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.14","timestamp":"2021-02-11T00:33:17.699242273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L43RwNuj0ddF9ifaSHgIKye","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.701224_81.1","section":"bucket","name":"ufpbfn-14","timestamp":"2021-02-11T00:33:17.701224321Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MRG403-5B5xPKMkhFWKEsyk","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.706794_82.1","section":"bucket","name":"ufpbfn-14","timestamp":"2021-02-11T00:33:17.706794690Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MRG403-5B5xPKMkhFWKEsyk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.754649_83.1","section":"bucket.instance","name":"ufpbfn-15:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.15","timestamp":"2021-02-11T00:33:17.754649827Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KrcZ_amVznZV1n91M3anyap","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.765027_84.1","section":"bucket.instance","name":"ufpbfn-15:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.15","timestamp":"2021-02-11T00:33:17.765027652Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KrcZ_amVznZV1n91M3anyap","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.767014_85.1","section":"bucket","name":"ufpbfn-15","timestamp":"2021-02-11T00:33:17.767014291Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EgRyHS2CuJZW4_js4EagF7N","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.771457_86.1","section":"bucket","name":"ufpbfn-15","timestamp":"2021-02-11T00:33:17.771457040Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EgRyHS2CuJZW4_js4EagF7N","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.388153_87.1","section":"bucket.instance","name":"ufpbfn-16:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.16","timestamp":"2021-02-11T00:33:26.388153899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.394315_93.1","section":"bucket.instance","name":"ufpbfn-16:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.16","timestamp":"2021-02-11T00:33:26.394315586Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.396589_94.1","section":"bucket","name":"ufpbfn-16","timestamp":"2021-02-11T00:33:26.396589995Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c6TEuYH8lTtdHiiKB2PN7B3","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.401860_95.1","section":"bucket","name":"ufpbfn-16","timestamp":"2021-02-11T00:33:26.401860934Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c6TEuYH8lTtdHiiKB2PN7B3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.426046_96.1","section":"bucket.instance","name":"ufpbfn-17:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.17","timestamp":"2021-02-11T00:33:26.426046415Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.431552_97.1","section":"bucket.instance","name":"ufpbfn-17:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.17","timestamp":"2021-02-11T00:33:26.431552117Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.433354_98.1","section":"bucket","name":"ufpbfn-17","timestamp":"2021-02-11T00:33:26.433354162Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6elO0Q-rH28Qr3aVT2qAwx6","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.438176_99.1","section":"bucket","name":"ufpbfn-17","timestamp":"2021-02-11T00:33:26.438176837Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6elO0Q-rH28Qr3aVT2qAwx6","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.480439_101.1","section":"bucket.instance","name":"ufpbfn-18:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.18","timestamp":"2021-02-11T00:33:26.480439651Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.486235_102.1","section":"bucket.instance","name":"ufpbfn-18:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.18","timestamp":"2021-02-11T00:33:26.486235046Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.488060_103.1","section":"bucket","name":"ufpbfn-18","timestamp":"2021-02-11T00:33:26.488060035Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s4Ba4emrAs-SlpdvF9nVDgT","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.492951_104.1","section":"bucket","name":"ufpbfn-18","timestamp":"2021-02-11T00:33:26.492951271Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s4Ba4emrAs-SlpdvF9nVDgT","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.515036_105.1","section":"bucket.instance","name":"ufpbfn-16:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.16","timestamp":"2021-02-11T00:33:26.515036772Z","data":{"read_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":1},"write_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":2},"status":{"status":"write"}}},{"id":"1_1613003606.520310_106.1","section":"bucket.instance","name":"ufpbfn-16:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.16","timestamp":"2021-02-11T00:33:26.520310165Z","data":{"read_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":1},"write_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003606.525055_107.1","section":"bucket.instance","name":"ufpbfn-17:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.17","timestamp":"2021-02-11T00:33:26.525055588Z","data":{"read_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":1},"write_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":2},"status":{"status":"write"}}},{"id":"1_1613003606.531303_108.1","section":"bucket.instance","name":"ufpbfn-17:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.17","timestamp":"2021-02-11T00:33:26.531303087Z","data":{"read_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":1},"write_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003606.542749_110.1","section":"bucket.instance","name":"ufpbfn-18:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.18","timestamp":"2021-02-11T00:33:26.542749975Z","data":{"read_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":1},"write_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":2},"status":{"status":"write"}}},{"id":"1_1613003606.555218_111.1","section":"bucket.instance","name":"ufpbfn-18:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.18","timestamp":"2021-02-11T00:33:26.555218578Z","data":{"read_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":1},"write_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003663.038735_296.1","section":"bucket.instance","name":"ufpbfn-19:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.19","timestamp":"2021-02-11T00:34:23.038735977Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":1},"status":{"status":"write"}}},{"id":"1_1613003663.044326_299.1","section":"bucket.instance","name":"ufpbfn-19:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.19","timestamp":"2021-02-11T00:34:23.044326194Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003663.046352_300.1","section":"bucket","name":"ufpbfn-19","timestamp":"2021-02-11T00:34:23.046352692Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_quU3_y1npE6yjsgMfaiPzD-","ver":1},"status":{"status":"write"}}},{"id":"1_1613003663.051849_301.1","section":"bucket","name":"ufpbfn-19","timestamp":"2021-02-11T00:34:23.051849389Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_quU3_y1npE6yjsgMfaiPzD-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003663.058645_302.1","section":"bucket.instance","name":"ufpbfn-19:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.19","timestamp":"2021-02-11T00:34:23.058645404Z","data":{"read_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":1},"write_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":2},"status":{"status":"write"}}},{"id":"1_1613003663.066081_303.1","section":"bucket.instance","name":"ufpbfn-19:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.19","timestamp":"2021-02-11T00:34:23.066081205Z","data":{"read_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":1},"write_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003710.698748_304.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:10.698748192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":1},"status":{"status":"write"}}},{"id":"1_1613003710.704651_318.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:10.704651509Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003710.706704_319.1","section":"bucket","name":"ufpbfn-20","timestamp":"2021-02-11T00:35:10.706704347Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OoZxlXdV7j7UCHfz9fjr3nn","ver":1},"status":{"status":"write"}}},{"id":"1_1613003710.715573_320.1","section":"bucket","name":"ufpbfn-20","timestamp":"2021-02-11T00:35:10.715573590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OoZxlXdV7j7UCHfz9fjr3nn","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003719.828914_353.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:19.828914598Z","data":{"read_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":1},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":2},"status":{"status":"write"}}},{"id":"1_1613003719.838940_354.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:19.838940378Z","data":{"read_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":1},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003726.262512_399.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:26.262512788Z","data":{"read_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":2},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":3},"status":{"status":"write"}}},{"id":"1_1613003726.272815_400.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:26.272815115Z","data":{"read_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":2},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":3},"status":{"status":"complete"}}},{"id":"1_1613003730.762664_445.1","section":"bucket.instance","name":"ufpbfn-21:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.21","timestamp":"2021-02-11T00:35:30.762664019Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.772882_446.1","section":"bucket.instance","name":"ufpbfn-21:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.21","timestamp":"2021-02-11T00:35:30.772882870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.775134_447.1","section":"bucket","name":"ufpbfn-21","timestamp":"2021-02-11T00:35:30.775134980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k5uY3Gk97MZZN-lG6Hb8Fye","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.783979_448.1","section":"bucket","name":"ufpbfn-21","timestamp":"2021-02-11T00:35:30.783979099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k5uY3Gk97MZZN-lG6Hb8Fye","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.811133_449.1","section":"bucket.instance","name":"ufpbfn-22:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.22","timestamp":"2021-02-11T00:35:30.811133053Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.816622_450.1","section":"bucket.instance","name":"ufpbfn-22:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.22","timestamp":"2021-02-11T00:35:30.816622426Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.818531_451.1","section":"bucket","name":"ufpbfn-22","timestamp":"2021-02-11T00:35:30.818531149Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FC7QfdFkMCENzYwmhnYnLvS","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.823109_452.1","section":"bucket","name":"ufpbfn-22","timestamp":"2021-02-11T00:35:30.823109099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FC7QfdFkMCENzYwmhnYnLvS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.866148_453.1","section":"bucket.instance","name":"ufpbfn-23:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.23","timestamp":"2021-02-11T00:35:30.866148114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.871906_454.1","section":"bucket.instance","name":"ufpbfn-23:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.23","timestamp":"2021-02-11T00:35:30.871906109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.875362_455.1","section":"bucket","name":"ufpbfn-23","timestamp":"2021-02-11T00:35:30.875362846Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JXU6nBLcF_sOtZUty9mXONE","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.880899_456.1","section":"bucket","name":"ufpbfn-23","timestamp":"2021-02-11T00:35:30.880899190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JXU6nBLcF_sOtZUty9mXONE","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.906538_457.1","section":"bucket.instance","name":"ufpbfn-21:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.21","timestamp":"2021-02-11T00:35:30.906538677Z","data":{"read_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":1},"write_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":2},"status":{"status":"write"}}},{"id":"1_1613003730.912385_458.1","section":"bucket.instance","name":"ufpbfn-21:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.21","timestamp":"2021-02-11T00:35:30.912385378Z","data":{"read_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":1},"write_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003730.916595_459.1","section":"bucket.instance","name":"ufpbfn-22:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.22","timestamp":"2021-02-11T00:35:30.916595556Z","data":{"read_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":1},"write_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":2},"status":{"status":"write"}}},{"id":"1_1613003730.921350_460.1","section":"bucket.instance","name":"ufpbfn-22:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.22","timestamp":"2021-02-11T00:35:30.921350521Z","data":{"read_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":1},"write_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003730.930587_461.1","section":"bucket.instance","name":"ufpbfn-23:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.23","timestamp":"2021-02-11T00:35:30.930587206Z","data":{"read_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":1},"write_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":2},"status":{"status":"write"}}},{"id":"1_1613003730.936764_462.1","section":"bucket.instance","name":"ufpbfn-23:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.23","timestamp":"2021-02-11T00:35:30.936764259Z","data":{"read_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":1},"write_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003742.361724_463.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.361724903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.367316_471.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.367316098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.369891_472.1","section":"bucket","name":"ufpbfn-24","timestamp":"2021-02-11T00:35:42.369891133Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3K3JZc4KLLW9-gE3essIbqt","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.375942_473.1","section":"bucket","name":"ufpbfn-24","timestamp":"2021-02-11T00:35:42.375942599Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3K3JZc4KLLW9-gE3essIbqt","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.404482_474.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.404482219Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.410336_475.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.410336039Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.412835_476.1","section":"bucket","name":"ufpbfn-25","timestamp":"2021-02-11T00:35:42.412835404Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a-6lD0m5KM_3p3bHNRp9pQd","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.417918_477.1","section":"bucket","name":"ufpbfn-25","timestamp":"2021-02-11T00:35:42.417918083Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a-6lD0m5KM_3p3bHNRp9pQd","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.451495_478.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.451495596Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.458335_479.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.458335827Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.463725_480.1","section":"bucket","name":"ufpbfn-26","timestamp":"2021-02-11T00:35:42.463725770Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_weJpVYgvsyih4Mc-CA_rT6D","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.471367_481.1","section":"bucket","name":"ufpbfn-26","timestamp":"2021-02-11T00:35:42.471367127Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_weJpVYgvsyih4Mc-CA_rT6D","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.492942_482.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.492942914Z","data":{"read_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":1},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":2},"status":{"status":"write"}}},{"id":"1_1613003742.498602_483.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.498602849Z","data":{"read_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":1},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003742.502980_484.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.502980249Z","data":{"read_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":2},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":3},"status":{"status":"write"}}},{"id":"1_1613003742.509352_485.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.509352182Z","data":{"read_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":2},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":3},"status":{"status":"complete"}}},{"id":"1_1613003742.515116_486.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.515116187Z","data":{"read_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":1},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":2},"status":{"status":"write"}}},{"id":"1_1613003742.522854_487.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.522854749Z","data":{"read_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":1},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003742.535194_488.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.535194409Z","data":{"read_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":2},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":3},"status":{"status":"write"}}},{"id":"1_1613003742.544167_489.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.544167180Z","data":{"read_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":2},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":3},"status":{"status":"complete"}}},{"id":"1_1613003742.556349_490.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.556349382Z","data":{"read_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":1},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":2},"status":{"status":"write"}}},{"id":"1_1613003742.566275_491.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.566275024Z","data":{"read_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":1},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003742.578851_492.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.578851542Z","data":{"read_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":2},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":3},"status":{"status":"write"}}},{"id":"1_1613003742.584739_493.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.584739159Z","data":{"read_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":2},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":3},"status":{"status":"complete"}}},{"id":"1_1613003752.114576_494.1","section":"bucket.instance","name":"ufpbfn-27:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.27","timestamp":"2021-02-11T00:35:52.114576760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.120282_502.1","section":"bucket.instance","name":"ufpbfn-27:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.27","timestamp":"2021-02-11T00:35:52.120282248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.122353_503.1","section":"bucket","name":"ufpbfn-27","timestamp":"2021-02-11T00:35:52.122353726Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wtfPlNxQltZmQNvUExRJKda","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.127478_504.1","section":"bucket","name":"ufpbfn-27","timestamp":"2021-02-11T00:35:52.127478495Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wtfPlNxQltZmQNvUExRJKda","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.150327_505.1","section":"bucket.instance","name":"ufpbfn-28:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.28","timestamp":"2021-02-11T00:35:52.150327964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.156621_506.1","section":"bucket.instance","name":"ufpbfn-28:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.28","timestamp":"2021-02-11T00:35:52.156621989Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.158759_507.1","section":"bucket","name":"ufpbfn-28","timestamp":"2021-02-11T00:35:52.158759219Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yn9gTjGHQR-RBwC8BZeqo8y","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.164363_508.1","section":"bucket","name":"ufpbfn-28","timestamp":"2021-02-11T00:35:52.164363564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yn9gTjGHQR-RBwC8BZeqo8y","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.210610_509.1","section":"bucket.instance","name":"ufpbfn-29:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.29","timestamp":"2021-02-11T00:35:52.210610079Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.218162_510.1","section":"bucket.instance","name":"ufpbfn-29:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.29","timestamp":"2021-02-11T00:35:52.218162393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.221895_511.1","section":"bucket","name":"ufpbfn-29","timestamp":"2021-02-11T00:35:52.221895004Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JVPKNH1TNGfzqm12utWovVB","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.232622_512.1","section":"bucket","name":"ufpbfn-29","timestamp":"2021-02-11T00:35:52.232622253Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JVPKNH1TNGfzqm12utWovVB","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.255266_513.1","section":"bucket.instance","name":"ufpbfn-27:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.27","timestamp":"2021-02-11T00:35:52.255266759Z","data":{"read_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":1},"write_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.265160_514.1","section":"bucket.instance","name":"ufpbfn-27:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.27","timestamp":"2021-02-11T00:35:52.265160032Z","data":{"read_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":1},"write_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.315161_515.1","section":"bucket.instance","name":"ufpbfn-28:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.28","timestamp":"2021-02-11T00:35:52.315161773Z","data":{"read_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":1},"write_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.325134_516.1","section":"bucket.instance","name":"ufpbfn-28:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.28","timestamp":"2021-02-11T00:35:52.325134952Z","data":{"read_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":1},"write_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.382688_517.1","section":"bucket.instance","name":"ufpbfn-29:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.29","timestamp":"2021-02-11T00:35:52.382688910Z","data":{"read_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":1},"write_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.393754_518.1","section":"bucket.instance","name":"ufpbfn-29:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.29","timestamp":"2021-02-11T00:35:52.393754381Z","data":{"read_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":1},"write_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.469936_519.1","section":"bucket.instance","name":"ufpbfn-30:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.30","timestamp":"2021-02-11T00:35:52.469936591Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.475466_520.1","section":"bucket.instance","name":"ufpbfn-30:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.30","timestamp":"2021-02-11T00:35:52.475466058Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.477315_521.1","section":"bucket","name":"ufpbfn-30","timestamp":"2021-02-11T00:35:52.477315559Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jjRH-goef-lAu1Abk5YXeix","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.481744_522.1","section":"bucket","name":"ufpbfn-30","timestamp":"2021-02-11T00:35:52.481744078Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jjRH-goef-lAu1Abk5YXeix","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.502453_523.1","section":"bucket.instance","name":"ufpbfn-31:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.31","timestamp":"2021-02-11T00:35:52.502453639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.508136_524.1","section":"bucket.instance","name":"ufpbfn-31:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.31","timestamp":"2021-02-11T00:35:52.508136124Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.510150_525.1","section":"bucket","name":"ufpbfn-31","timestamp":"2021-02-11T00:35:52.510150885Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YkTpeSeHlvxSYw5Rw8Ab4_g","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.515456_526.1","section":"bucket","name":"ufpbfn-31","timestamp":"2021-02-11T00:35:52.515456566Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YkTpeSeHlvxSYw5Rw8Ab4_g","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.561311_527.1","section":"bucket.instance","name":"ufpbfn-32:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.32","timestamp":"2021-02-11T00:35:52.561311410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.567433_528.1","section":"bucket.instance","name":"ufpbfn-32:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.32","timestamp":"2021-02-11T00:35:52.567433316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.569487_529.1","section":"bucket","name":"ufpbfn-32","timestamp":"2021-02-11T00:35:52.569487932Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LHW-hTrz97v-1TYWLUyydCX","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.574147_530.1","section":"bucket","name":"ufpbfn-32","timestamp":"2021-02-11T00:35:52.574147158Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LHW-hTrz97v-1TYWLUyydCX","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.597912_531.1","section":"bucket.instance","name":"ufpbfn-30:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.30","timestamp":"2021-02-11T00:35:52.597912475Z","data":{"read_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":1},"write_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.604469_532.1","section":"bucket.instance","name":"ufpbfn-30:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.30","timestamp":"2021-02-11T00:35:52.604469492Z","data":{"read_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":1},"write_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.656328_533.1","section":"bucket.instance","name":"ufpbfn-31:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.31","timestamp":"2021-02-11T00:35:52.656328101Z","data":{"read_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":1},"write_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.668788_534.1","section":"bucket.instance","name":"ufpbfn-31:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.31","timestamp":"2021-02-11T00:35:52.668788500Z","data":{"read_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":1},"write_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.724450_535.1","section":"bucket.instance","name":"ufpbfn-32:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.32","timestamp":"2021-02-11T00:35:52.724450815Z","data":{"read_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":1},"write_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.732591_536.1","section":"bucket.instance","name":"ufpbfn-32:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.32","timestamp":"2021-02-11T00:35:52.732591666Z","data":{"read_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":1},"write_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.808374_537.1","section":"bucket.instance","name":"ufpbfn-33:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.33","timestamp":"2021-02-11T00:35:52.808374402Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.815859_538.1","section":"bucket.instance","name":"ufpbfn-33:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.33","timestamp":"2021-02-11T00:35:52.815859569Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.821293_539.1","section":"bucket","name":"ufpbfn-33","timestamp":"2021-02-11T00:35:52.821293819Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qyZG5IHPnJ0q7XL8oHBFG_M","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.833861_540.1","section":"bucket","name":"ufpbfn-33","timestamp":"2021-02-11T00:35:52.833861919Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qyZG5IHPnJ0q7XL8oHBFG_M","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.855079_541.1","section":"bucket.instance","name":"ufpbfn-34:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.34","timestamp":"2021-02-11T00:35:52.855079273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.863679_542.1","section":"bucket.instance","name":"ufpbfn-34:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.34","timestamp":"2021-02-11T00:35:52.863679302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.865715_543.1","section":"bucket","name":"ufpbfn-34","timestamp":"2021-02-11T00:35:52.865715392Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eaPmsUe9UpKZde4q81ZEGn-","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.870203_544.1","section":"bucket","name":"ufpbfn-34","timestamp":"2021-02-11T00:35:52.870203091Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eaPmsUe9UpKZde4q81ZEGn-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.911282_545.1","section":"bucket.instance","name":"ufpbfn-35:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.35","timestamp":"2021-02-11T00:35:52.911282623Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.917850_546.1","section":"bucket.instance","name":"ufpbfn-35:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.35","timestamp":"2021-02-11T00:35:52.917850358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.920196_547.1","section":"bucket","name":"ufpbfn-35","timestamp":"2021-02-11T00:35:52.920196626Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2P7hUA6fxJFof13fab6ELfJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.925674_548.1","section":"bucket","name":"ufpbfn-35","timestamp":"2021-02-11T00:35:52.925674365Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2P7hUA6fxJFof13fab6ELfJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.946987_549.1","section":"bucket.instance","name":"ufpbfn-33:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.33","timestamp":"2021-02-11T00:35:52.946987783Z","data":{"read_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":1},"write_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.952944_550.1","section":"bucket.instance","name":"ufpbfn-33:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.33","timestamp":"2021-02-11T00:35:52.952944552Z","data":{"read_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":1},"write_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003754.007637_551.1","section":"bucket.instance","name":"ufpbfn-34:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.34","timestamp":"2021-02-11T00:35:54.007637789Z","data":{"read_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":1},"write_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":2},"status":{"status":"write"}}},{"id":"1_1613003754.017772_552.1","section":"bucket.instance","name":"ufpbfn-34:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.34","timestamp":"2021-02-11T00:35:54.017772278Z","data":{"read_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":1},"write_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003755.079943_553.1","section":"bucket.instance","name":"ufpbfn-35:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.35","timestamp":"2021-02-11T00:35:55.079943426Z","data":{"read_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":1},"write_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":2},"status":{"status":"write"}}},{"id":"1_1613003755.092005_554.1","section":"bucket.instance","name":"ufpbfn-35:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.35","timestamp":"2021-02-11T00:35:55.092005034Z","data":{"read_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":1},"write_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003755.170194_555.1","section":"bucket.instance","name":"ufpbfn-36:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.36","timestamp":"2021-02-11T00:35:55.170194935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6Bhkdo_dUOAdMeAbwoirslC","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.177626_556.1","section":"bucket.instance","name":"ufpbfn-36:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.36","timestamp":"2021-02-11T00:35:55.177626644Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6Bhkdo_dUOAdMeAbwoirslC","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.180328_557.1","section":"bucket","name":"ufpbfn-36","timestamp":"2021-02-11T00:35:55.180328782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yhkND0d6ef1KagNiHqzkC3_","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.186832_558.1","section":"bucket","name":"ufpbfn-36","timestamp":"2021-02-11T00:35:55.186832130Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yhkND0d6ef1KagNiHqzkC3_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.224261_559.1","section":"bucket.instance","name":"ufpbfn-37:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.37","timestamp":"2021-02-11T00:35:55.224261590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j2fBBQD3zN4CmLC7L14TrCt","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.232953_560.1","section":"bucket.instance","name":"ufpbfn-37:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.37","timestamp":"2021-02-11T00:35:55.232953898Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j2fBBQD3zN4CmLC7L14TrCt","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.235184_561.1","section":"bucket","name":"ufpbfn-37","timestamp":"2021-02-11T00:35:55.235184111Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ET2FdSdbJ8ZEOCAPZJ-po8a","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.239904_562.1","section":"bucket","name":"ufpbfn-37","timestamp":"2021-02-11T00:35:55.239904921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ET2FdSdbJ8ZEOCAPZJ-po8a","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.279481_563.1","section":"bucket.instance","name":"ufpbfn-38:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.38","timestamp":"2021-02-11T00:35:55.279481119Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DmS3IetBwqoZFY34Xk3Ohek","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.284473_564.1","section":"bucket.instance","name":"ufpbfn-38:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.38","timestamp":"2021-02-11T00:35:55.284473066Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DmS3IetBwqoZFY34Xk3Ohek","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.286540_565.1","section":"bucket","name":"ufpbfn-38","timestamp":"2021-02-11T00:35:55.286540135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0ru7BE0bzAXx0-Pb22dsKmU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.291866_566.1","section":"bucket","name":"ufpbfn-38","timestamp":"2021-02-11T00:35:55.291866653Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0ru7BE0bzAXx0-Pb22dsKmU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.788416_570.1","section":"bucket.instance","name":"ufpbfn-39:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.39","timestamp":"2021-02-11T00:35:57.788416591Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1vgWQA2CxVJU7vO6qXuqW2w","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.796322_571.1","section":"bucket.instance","name":"ufpbfn-39:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.39","timestamp":"2021-02-11T00:35:57.796322629Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1vgWQA2CxVJU7vO6qXuqW2w","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.798608_572.1","section":"bucket","name":"ufpbfn-39","timestamp":"2021-02-11T00:35:57.798608517Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kSxSKKo8ruuBEqht7yFAiKh","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.804310_573.1","section":"bucket","name":"ufpbfn-39","timestamp":"2021-02-11T00:35:57.804310795Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kSxSKKo8ruuBEqht7yFAiKh","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.834923_574.1","section":"bucket.instance","name":"ufpbfn-40:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.40","timestamp":"2021-02-11T00:35:57.834923842Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jcrlr-QMPmHhG1khCqUK-HL","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.844273_575.1","section":"bucket.instance","name":"ufpbfn-40:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.40","timestamp":"2021-02-11T00:35:57.844273207Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jcrlr-QMPmHhG1khCqUK-HL","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.846725_576.1","section":"bucket","name":"ufpbfn-40","timestamp":"2021-02-11T00:35:57.846725006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-U2oP92YIGzVDnZTkauQDfU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.851372_577.1","section":"bucket","name":"ufpbfn-40","timestamp":"2021-02-11T00:35:57.851372809Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-U2oP92YIGzVDnZTkauQDfU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.893851_578.1","section":"bucket.instance","name":"ufpbfn-41:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.41","timestamp":"2021-02-11T00:35:57.893851494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_F6KHu-saDEKBY8ttlAXOWyy","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.902224_579.1","section":"bucket.instance","name":"ufpbfn-41:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.41","timestamp":"2021-02-11T00:35:57.902224837Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_F6KHu-saDEKBY8ttlAXOWyy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.904331_580.1","section":"bucket","name":"ufpbfn-41","timestamp":"2021-02-11T00:35:57.904331122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Hr9JbpKcJAgiTFrG2I388EO","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.913330_581.1","section":"bucket","name":"ufpbfn-41","timestamp":"2021-02-11T00:35:57.913330476Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Hr9JbpKcJAgiTFrG2I388EO","ver":1},"status":{"status":"complete"}}}] 2021-02-11T00:38:16.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '7f3547bf-73e3-43a7-aa91-648db0f982f6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:16.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '7f3547bf-73e3-43a7-aa91-648db0f982f6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:16.091 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 7f3547bf-73e3-43a7-aa91-648db0f982f6 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:16.131 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:16.131 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:16.396 INFO:teuthology.orchestra.run.smithi057.stdout:[{"id":"1_1613003450.854302_44.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T00:30:50.854302550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8afXW5L0598IonMpxK19QLU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003450.863150_45.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T00:30:50.863150983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8afXW5L0598IonMpxK19QLU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003573.921741_48.1","section":"bucket.instance","name":"ufpbfn-1:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.1","timestamp":"2021-02-11T00:32:53.921741821Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Skjph_QFacdJPxfoZZsFlwv","ver":1},"status":{"status":"write"}}},{"id":"1_1613003573.938336_49.1","section":"bucket.instance","name":"ufpbfn-1:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.1","timestamp":"2021-02-11T00:32:53.938336161Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Skjph_QFacdJPxfoZZsFlwv","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003573.939868_50.1","section":"bucket","name":"ufpbfn-1","timestamp":"2021-02-11T00:32:53.939868855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KqGKnzK6Tf8QvJEeDxt1_WH","ver":1},"status":{"status":"write"}}},{"id":"1_1613003573.944120_51.1","section":"bucket","name":"ufpbfn-1","timestamp":"2021-02-11T00:32:53.944120215Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KqGKnzK6Tf8QvJEeDxt1_WH","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003573.965387_52.1","section":"bucket.instance","name":"ufpbfn-2:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.2","timestamp":"2021-02-11T00:32:53.965387242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_opo5r9iAQSy3iT3OqymSbUR","ver":1},"status":{"status":"write"}}},{"id":"1_1613003573.970287_53.1","section":"bucket.instance","name":"ufpbfn-2:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.2","timestamp":"2021-02-11T00:32:53.970287398Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_opo5r9iAQSy3iT3OqymSbUR","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003573.971912_54.1","section":"bucket","name":"ufpbfn-2","timestamp":"2021-02-11T00:32:53.971912556Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I8Y2RjNUm2--MMKap47mPvi","ver":1},"status":{"status":"write"}}},{"id":"1_1613003573.976646_55.1","section":"bucket","name":"ufpbfn-2","timestamp":"2021-02-11T00:32:53.976646634Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I8Y2RjNUm2--MMKap47mPvi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003577.130440_56.1","section":"bucket.instance","name":"ufpbfn-3:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.3","timestamp":"2021-02-11T00:32:57.130440180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__JIYMfX43-rh93xo8qCfW3W","ver":1},"status":{"status":"write"}}},{"id":"1_1613003577.138907_57.1","section":"bucket.instance","name":"ufpbfn-3:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.3","timestamp":"2021-02-11T00:32:57.138907718Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__JIYMfX43-rh93xo8qCfW3W","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003577.141172_58.1","section":"bucket","name":"ufpbfn-3","timestamp":"2021-02-11T00:32:57.141172704Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZD69Kxj7MfN4ykEqn2AmMHP","ver":1},"status":{"status":"write"}}},{"id":"1_1613003577.148691_59.1","section":"bucket","name":"ufpbfn-3","timestamp":"2021-02-11T00:32:57.148691001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZD69Kxj7MfN4ykEqn2AmMHP","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.351288_60.1","section":"bucket.instance","name":"ufpbfn-4:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.4","timestamp":"2021-02-11T00:32:58.351288084Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7b_KGvalODv_9Uf9Kk5icn8","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.356570_61.1","section":"bucket.instance","name":"ufpbfn-4:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.4","timestamp":"2021-02-11T00:32:58.356570584Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7b_KGvalODv_9Uf9Kk5icn8","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.358632_62.1","section":"bucket","name":"ufpbfn-4","timestamp":"2021-02-11T00:32:58.358632530Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jr7oDJUzCTW6gHPoeufR7jm","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.363794_63.1","section":"bucket","name":"ufpbfn-4","timestamp":"2021-02-11T00:32:58.363794123Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jr7oDJUzCTW6gHPoeufR7jm","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.382871_64.1","section":"bucket.instance","name":"ufpbfn-5:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.5","timestamp":"2021-02-11T00:32:58.382871708Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a0Y62ZJ3YMY_wT-paa_QHFa","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.388437_65.1","section":"bucket.instance","name":"ufpbfn-5:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.5","timestamp":"2021-02-11T00:32:58.388437173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a0Y62ZJ3YMY_wT-paa_QHFa","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.390530_66.1","section":"bucket","name":"ufpbfn-5","timestamp":"2021-02-11T00:32:58.390530916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZMPDqfQwJlwt4fOPIl0M8aR","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.395610_67.1","section":"bucket","name":"ufpbfn-5","timestamp":"2021-02-11T00:32:58.395610305Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZMPDqfQwJlwt4fOPIl0M8aR","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.436947_68.1","section":"bucket.instance","name":"ufpbfn-6:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.6","timestamp":"2021-02-11T00:32:58.436947070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iVj6C98NL7g1XpDBkqjdwxs","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.442139_69.1","section":"bucket.instance","name":"ufpbfn-6:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.6","timestamp":"2021-02-11T00:32:58.442139964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iVj6C98NL7g1XpDBkqjdwxs","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.443881_70.1","section":"bucket","name":"ufpbfn-6","timestamp":"2021-02-11T00:32:58.443881886Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gm9dBc_bz3ThL3EzllaIRon","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.448531_71.1","section":"bucket","name":"ufpbfn-6","timestamp":"2021-02-11T00:32:58.448531161Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gm9dBc_bz3ThL3EzllaIRon","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.032175_72.1","section":"bucket.instance","name":"ufpbfn-7:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.7","timestamp":"2021-02-11T00:33:01.032175874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6WrCP2fZ4q43uXb3VOBaP-X","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.037898_73.1","section":"bucket.instance","name":"ufpbfn-7:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.7","timestamp":"2021-02-11T00:33:01.037898478Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6WrCP2fZ4q43uXb3VOBaP-X","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.040379_74.1","section":"bucket","name":"ufpbfn-7","timestamp":"2021-02-11T00:33:01.040379029Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.046480_75.1","section":"bucket","name":"ufpbfn-7","timestamp":"2021-02-11T00:33:01.046480658Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.066133_76.1","section":"bucket.instance","name":"ufpbfn-8:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.8","timestamp":"2021-02-11T00:33:01.066133971Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ImgVKlW5BV_4yXJ5eaN1blE","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.071786_77.1","section":"bucket.instance","name":"ufpbfn-8:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.8","timestamp":"2021-02-11T00:33:01.071786138Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ImgVKlW5BV_4yXJ5eaN1blE","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.073784_78.1","section":"bucket","name":"ufpbfn-8","timestamp":"2021-02-11T00:33:01.073784230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.078439_79.1","section":"bucket","name":"ufpbfn-8","timestamp":"2021-02-11T00:33:01.078439230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.118285_80.1","section":"bucket.instance","name":"ufpbfn-9:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.9","timestamp":"2021-02-11T00:33:01.118285107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4WCKMP2jWFDUs3dtJoGmuxJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.123404_81.1","section":"bucket.instance","name":"ufpbfn-9:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.9","timestamp":"2021-02-11T00:33:01.123404230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4WCKMP2jWFDUs3dtJoGmuxJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.125274_82.1","section":"bucket","name":"ufpbfn-9","timestamp":"2021-02-11T00:33:01.125274841Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.130865_83.1","section":"bucket","name":"ufpbfn-9","timestamp":"2021-02-11T00:33:01.130865699Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003582.332946_84.1","section":"bucket","name":"ufpbfn-7","timestamp":"2021-02-11T00:33:02.332946911Z","data":{"read_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":1},"write_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":2},"status":{"status":"remove"}}},{"id":"1_1613003582.340020_85.1","section":"bucket","name":"ufpbfn-7","timestamp":"2021-02-11T00:33:02.340020754Z","data":{"read_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":1},"write_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003582.420876_86.1","section":"bucket","name":"ufpbfn-8","timestamp":"2021-02-11T00:33:02.420876072Z","data":{"read_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":1},"write_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":2},"status":{"status":"remove"}}},{"id":"1_1613003582.427710_87.1","section":"bucket","name":"ufpbfn-8","timestamp":"2021-02-11T00:33:02.427710584Z","data":{"read_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":1},"write_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003582.554110_88.1","section":"bucket","name":"ufpbfn-9","timestamp":"2021-02-11T00:33:02.554110401Z","data":{"read_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":1},"write_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":2},"status":{"status":"remove"}}},{"id":"1_1613003582.559316_89.1","section":"bucket","name":"ufpbfn-9","timestamp":"2021-02-11T00:33:02.559316632Z","data":{"read_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":1},"write_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003583.730202_48.1","section":"bucket.instance","name":"ufpbfn-10:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.10","timestamp":"2021-02-11T00:33:03.730202256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2eGqXaG3AOEn7phUs2DcOIc","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.736098_58.1","section":"bucket.instance","name":"ufpbfn-10:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.10","timestamp":"2021-02-11T00:33:03.736098882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2eGqXaG3AOEn7phUs2DcOIc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.738359_59.1","section":"bucket","name":"ufpbfn-10","timestamp":"2021-02-11T00:33:03.738359979Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tDxlkqHxmS7PTDEab2H7hMT","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.744305_60.1","section":"bucket","name":"ufpbfn-10","timestamp":"2021-02-11T00:33:03.744305538Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tDxlkqHxmS7PTDEab2H7hMT","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.767444_61.1","section":"bucket.instance","name":"ufpbfn-11:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.11","timestamp":"2021-02-11T00:33:03.767444315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wkX6SG6rOg3K4s40k0F4qy5","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.773359_62.1","section":"bucket.instance","name":"ufpbfn-11:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.11","timestamp":"2021-02-11T00:33:03.773359088Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wkX6SG6rOg3K4s40k0F4qy5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.775379_63.1","section":"bucket","name":"ufpbfn-11","timestamp":"2021-02-11T00:33:03.775379051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U4YSB-MjK3Lwa5aeb8ZJsWC","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.780715_64.1","section":"bucket","name":"ufpbfn-11","timestamp":"2021-02-11T00:33:03.780715854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U4YSB-MjK3Lwa5aeb8ZJsWC","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.825241_65.1","section":"bucket.instance","name":"ufpbfn-12:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.12","timestamp":"2021-02-11T00:33:03.825241462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jq-H07HPoS-gctsMOEeF9RO","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.830032_66.1","section":"bucket.instance","name":"ufpbfn-12:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.12","timestamp":"2021-02-11T00:33:03.830032565Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jq-H07HPoS-gctsMOEeF9RO","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.832053_67.1","section":"bucket","name":"ufpbfn-12","timestamp":"2021-02-11T00:33:03.832053315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__fBfLcBSoKwIw5StU9Nvh6L","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.836924_68.1","section":"bucket","name":"ufpbfn-12","timestamp":"2021-02-11T00:33:03.836924032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__fBfLcBSoKwIw5StU9Nvh6L","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.659565_73.1","section":"bucket.instance","name":"ufpbfn-13:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.13","timestamp":"2021-02-11T00:33:17.659565432Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f8SIjk338xKgtCDgHkGKqff","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.665073_76.1","section":"bucket.instance","name":"ufpbfn-13:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.13","timestamp":"2021-02-11T00:33:17.665073710Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f8SIjk338xKgtCDgHkGKqff","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.666958_77.1","section":"bucket","name":"ufpbfn-13","timestamp":"2021-02-11T00:33:17.666958458Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0NQ3xjtUNQUmGx9jbXsmf2Z","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.671944_78.1","section":"bucket","name":"ufpbfn-13","timestamp":"2021-02-11T00:33:17.671944640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0NQ3xjtUNQUmGx9jbXsmf2Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.692682_79.1","section":"bucket.instance","name":"ufpbfn-14:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.14","timestamp":"2021-02-11T00:33:17.692682553Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L43RwNuj0ddF9ifaSHgIKye","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.699242_80.1","section":"bucket.instance","name":"ufpbfn-14:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.14","timestamp":"2021-02-11T00:33:17.699242273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L43RwNuj0ddF9ifaSHgIKye","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.701224_81.1","section":"bucket","name":"ufpbfn-14","timestamp":"2021-02-11T00:33:17.701224321Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MRG403-5B5xPKMkhFWKEsyk","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.706794_82.1","section":"bucket","name":"ufpbfn-14","timestamp":"2021-02-11T00:33:17.706794690Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MRG403-5B5xPKMkhFWKEsyk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.754649_83.1","section":"bucket.instance","name":"ufpbfn-15:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.15","timestamp":"2021-02-11T00:33:17.754649827Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KrcZ_amVznZV1n91M3anyap","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.765027_84.1","section":"bucket.instance","name":"ufpbfn-15:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.15","timestamp":"2021-02-11T00:33:17.765027652Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KrcZ_amVznZV1n91M3anyap","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.767014_85.1","section":"bucket","name":"ufpbfn-15","timestamp":"2021-02-11T00:33:17.767014291Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EgRyHS2CuJZW4_js4EagF7N","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.771457_86.1","section":"bucket","name":"ufpbfn-15","timestamp":"2021-02-11T00:33:17.771457040Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EgRyHS2CuJZW4_js4EagF7N","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.388153_87.1","section":"bucket.instance","name":"ufpbfn-16:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.16","timestamp":"2021-02-11T00:33:26.388153899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.394315_93.1","section":"bucket.instance","name":"ufpbfn-16:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.16","timestamp":"2021-02-11T00:33:26.394315586Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.396589_94.1","section":"bucket","name":"ufpbfn-16","timestamp":"2021-02-11T00:33:26.396589995Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c6TEuYH8lTtdHiiKB2PN7B3","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.401860_95.1","section":"bucket","name":"ufpbfn-16","timestamp":"2021-02-11T00:33:26.401860934Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c6TEuYH8lTtdHiiKB2PN7B3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.426046_96.1","section":"bucket.instance","name":"ufpbfn-17:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.17","timestamp":"2021-02-11T00:33:26.426046415Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.431552_97.1","section":"bucket.instance","name":"ufpbfn-17:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.17","timestamp":"2021-02-11T00:33:26.431552117Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.433354_98.1","section":"bucket","name":"ufpbfn-17","timestamp":"2021-02-11T00:33:26.433354162Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6elO0Q-rH28Qr3aVT2qAwx6","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.438176_99.1","section":"bucket","name":"ufpbfn-17","timestamp":"2021-02-11T00:33:26.438176837Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6elO0Q-rH28Qr3aVT2qAwx6","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.480439_101.1","section":"bucket.instance","name":"ufpbfn-18:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.18","timestamp":"2021-02-11T00:33:26.480439651Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.486235_102.1","section":"bucket.instance","name":"ufpbfn-18:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.18","timestamp":"2021-02-11T00:33:26.486235046Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.488060_103.1","section":"bucket","name":"ufpbfn-18","timestamp":"2021-02-11T00:33:26.488060035Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s4Ba4emrAs-SlpdvF9nVDgT","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.492951_104.1","section":"bucket","name":"ufpbfn-18","timestamp":"2021-02-11T00:33:26.492951271Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s4Ba4emrAs-SlpdvF9nVDgT","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.515036_105.1","section":"bucket.instance","name":"ufpbfn-16:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.16","timestamp":"2021-02-11T00:33:26.515036772Z","data":{"read_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":1},"write_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":2},"status":{"status":"write"}}},{"id":"1_1613003606.520310_106.1","section":"bucket.instance","name":"ufpbfn-16:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.16","timestamp":"2021-02-11T00:33:26.520310165Z","data":{"read_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":1},"write_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003606.525055_107.1","section":"bucket.instance","name":"ufpbfn-17:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.17","timestamp":"2021-02-11T00:33:26.525055588Z","data":{"read_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":1},"write_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":2},"status":{"status":"write"}}},{"id":"1_1613003606.531303_108.1","section":"bucket.instance","name":"ufpbfn-17:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.17","timestamp":"2021-02-11T00:33:26.531303087Z","data":{"read_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":1},"write_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003606.542749_110.1","section":"bucket.instance","name":"ufpbfn-18:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.18","timestamp":"2021-02-11T00:33:26.542749975Z","data":{"read_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":1},"write_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":2},"status":{"status":"write"}}},{"id":"1_1613003606.555218_111.1","section":"bucket.instance","name":"ufpbfn-18:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.18","timestamp":"2021-02-11T00:33:26.555218578Z","data":{"read_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":1},"write_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003663.038735_296.1","section":"bucket.instance","name":"ufpbfn-19:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.19","timestamp":"2021-02-11T00:34:23.038735977Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":1},"status":{"status":"write"}}},{"id":"1_1613003663.044326_299.1","section":"bucket.instance","name":"ufpbfn-19:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.19","timestamp":"2021-02-11T00:34:23.044326194Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003663.046352_300.1","section":"bucket","name":"ufpbfn-19","timestamp":"2021-02-11T00:34:23.046352692Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_quU3_y1npE6yjsgMfaiPzD-","ver":1},"status":{"status":"write"}}},{"id":"1_1613003663.051849_301.1","section":"bucket","name":"ufpbfn-19","timestamp":"2021-02-11T00:34:23.051849389Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_quU3_y1npE6yjsgMfaiPzD-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003663.058645_302.1","section":"bucket.instance","name":"ufpbfn-19:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.19","timestamp":"2021-02-11T00:34:23.058645404Z","data":{"read_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":1},"write_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":2},"status":{"status":"write"}}},{"id":"1_1613003663.066081_303.1","section":"bucket.instance","name":"ufpbfn-19:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.19","timestamp":"2021-02-11T00:34:23.066081205Z","data":{"read_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":1},"write_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003710.698748_304.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:10.698748192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":1},"status":{"status":"write"}}},{"id":"1_1613003710.704651_318.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:10.704651509Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003710.706704_319.1","section":"bucket","name":"ufpbfn-20","timestamp":"2021-02-11T00:35:10.706704347Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OoZxlXdV7j7UCHfz9fjr3nn","ver":1},"status":{"status":"write"}}},{"id":"1_1613003710.715573_320.1","section":"bucket","name":"ufpbfn-20","timestamp":"2021-02-11T00:35:10.715573590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OoZxlXdV7j7UCHfz9fjr3nn","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003719.828914_353.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:19.828914598Z","data":{"read_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":1},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":2},"status":{"status":"write"}}},{"id":"1_1613003719.838940_354.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:19.838940378Z","data":{"read_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":1},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003726.262512_399.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:26.262512788Z","data":{"read_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":2},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":3},"status":{"status":"write"}}},{"id":"1_1613003726.272815_400.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:26.272815115Z","data":{"read_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":2},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":3},"status":{"status":"complete"}}},{"id":"1_1613003730.762664_445.1","section":"bucket.instance","name":"ufpbfn-21:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.21","timestamp":"2021-02-11T00:35:30.762664019Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.772882_446.1","section":"bucket.instance","name":"ufpbfn-21:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.21","timestamp":"2021-02-11T00:35:30.772882870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.775134_447.1","section":"bucket","name":"ufpbfn-21","timestamp":"2021-02-11T00:35:30.775134980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k5uY3Gk97MZZN-lG6Hb8Fye","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.783979_448.1","section":"bucket","name":"ufpbfn-21","timestamp":"2021-02-11T00:35:30.783979099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k5uY3Gk97MZZN-lG6Hb8Fye","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.811133_449.1","section":"bucket.instance","name":"ufpbfn-22:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.22","timestamp":"2021-02-11T00:35:30.811133053Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.816622_450.1","section":"bucket.instance","name":"ufpbfn-22:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.22","timestamp":"2021-02-11T00:35:30.816622426Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.818531_451.1","section":"bucket","name":"ufpbfn-22","timestamp":"2021-02-11T00:35:30.818531149Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FC7QfdFkMCENzYwmhnYnLvS","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.823109_452.1","section":"bucket","name":"ufpbfn-22","timestamp":"2021-02-11T00:35:30.823109099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FC7QfdFkMCENzYwmhnYnLvS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.866148_453.1","section":"bucket.instance","name":"ufpbfn-23:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.23","timestamp":"2021-02-11T00:35:30.866148114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.871906_454.1","section":"bucket.instance","name":"ufpbfn-23:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.23","timestamp":"2021-02-11T00:35:30.871906109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.875362_455.1","section":"bucket","name":"ufpbfn-23","timestamp":"2021-02-11T00:35:30.875362846Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JXU6nBLcF_sOtZUty9mXONE","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.880899_456.1","section":"bucket","name":"ufpbfn-23","timestamp":"2021-02-11T00:35:30.880899190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JXU6nBLcF_sOtZUty9mXONE","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.906538_457.1","section":"bucket.instance","name":"ufpbfn-21:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.21","timestamp":"2021-02-11T00:35:30.906538677Z","data":{"read_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":1},"write_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":2},"status":{"status":"write"}}},{"id":"1_1613003730.912385_458.1","section":"bucket.instance","name":"ufpbfn-21:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.21","timestamp":"2021-02-11T00:35:30.912385378Z","data":{"read_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":1},"write_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003730.916595_459.1","section":"bucket.instance","name":"ufpbfn-22:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.22","timestamp":"2021-02-11T00:35:30.916595556Z","data":{"read_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":1},"write_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":2},"status":{"status":"write"}}},{"id":"1_1613003730.921350_460.1","section":"bucket.instance","name":"ufpbfn-22:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.22","timestamp":"2021-02-11T00:35:30.921350521Z","data":{"read_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":1},"write_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003730.930587_461.1","section":"bucket.instance","name":"ufpbfn-23:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.23","timestamp":"2021-02-11T00:35:30.930587206Z","data":{"read_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":1},"write_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":2},"status":{"status":"write"}}},{"id":"1_1613003730.936764_462.1","section":"bucket.instance","name":"ufpbfn-23:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.23","timestamp":"2021-02-11T00:35:30.936764259Z","data":{"read_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":1},"write_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003742.361724_463.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.361724903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.367316_471.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.367316098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.369891_472.1","section":"bucket","name":"ufpbfn-24","timestamp":"2021-02-11T00:35:42.369891133Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3K3JZc4KLLW9-gE3essIbqt","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.375942_473.1","section":"bucket","name":"ufpbfn-24","timestamp":"2021-02-11T00:35:42.375942599Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3K3JZc4KLLW9-gE3essIbqt","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.404482_474.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.404482219Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.410336_475.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.410336039Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.412835_476.1","section":"bucket","name":"ufpbfn-25","timestamp":"2021-02-11T00:35:42.412835404Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a-6lD0m5KM_3p3bHNRp9pQd","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.417918_477.1","section":"bucket","name":"ufpbfn-25","timestamp":"2021-02-11T00:35:42.417918083Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a-6lD0m5KM_3p3bHNRp9pQd","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.451495_478.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.451495596Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.458335_479.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.458335827Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.463725_480.1","section":"bucket","name":"ufpbfn-26","timestamp":"2021-02-11T00:35:42.463725770Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_weJpVYgvsyih4Mc-CA_rT6D","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.471367_481.1","section":"bucket","name":"ufpbfn-26","timestamp":"2021-02-11T00:35:42.471367127Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_weJpVYgvsyih4Mc-CA_rT6D","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.492942_482.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.492942914Z","data":{"read_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":1},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":2},"status":{"status":"write"}}},{"id":"1_1613003742.498602_483.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.498602849Z","data":{"read_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":1},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003742.502980_484.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.502980249Z","data":{"read_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":2},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":3},"status":{"status":"write"}}},{"id":"1_1613003742.509352_485.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.509352182Z","data":{"read_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":2},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":3},"status":{"status":"complete"}}},{"id":"1_1613003742.515116_486.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.515116187Z","data":{"read_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":1},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":2},"status":{"status":"write"}}},{"id":"1_1613003742.522854_487.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.522854749Z","data":{"read_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":1},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003742.535194_488.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.535194409Z","data":{"read_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":2},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":3},"status":{"status":"write"}}},{"id":"1_1613003742.544167_489.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.544167180Z","data":{"read_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":2},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":3},"status":{"status":"complete"}}},{"id":"1_1613003742.556349_490.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.556349382Z","data":{"read_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":1},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":2},"status":{"status":"write"}}},{"id":"1_1613003742.566275_491.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.566275024Z","data":{"read_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":1},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003742.578851_492.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.578851542Z","data":{"read_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":2},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":3},"status":{"status":"write"}}},{"id":"1_1613003742.584739_493.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.584739159Z","data":{"read_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":2},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":3},"status":{"status":"complete"}}},{"id":"1_1613003752.114576_494.1","section":"bucket.instance","name":"ufpbfn-27:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.27","timestamp":"2021-02-11T00:35:52.114576760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.120282_502.1","section":"bucket.instance","name":"ufpbfn-27:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.27","timestamp":"2021-02-11T00:35:52.120282248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.122353_503.1","section":"bucket","name":"ufpbfn-27","timestamp":"2021-02-11T00:35:52.122353726Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wtfPlNxQltZmQNvUExRJKda","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.127478_504.1","section":"bucket","name":"ufpbfn-27","timestamp":"2021-02-11T00:35:52.127478495Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wtfPlNxQltZmQNvUExRJKda","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.150327_505.1","section":"bucket.instance","name":"ufpbfn-28:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.28","timestamp":"2021-02-11T00:35:52.150327964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.156621_506.1","section":"bucket.instance","name":"ufpbfn-28:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.28","timestamp":"2021-02-11T00:35:52.156621989Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.158759_507.1","section":"bucket","name":"ufpbfn-28","timestamp":"2021-02-11T00:35:52.158759219Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yn9gTjGHQR-RBwC8BZeqo8y","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.164363_508.1","section":"bucket","name":"ufpbfn-28","timestamp":"2021-02-11T00:35:52.164363564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yn9gTjGHQR-RBwC8BZeqo8y","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.210610_509.1","section":"bucket.instance","name":"ufpbfn-29:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.29","timestamp":"2021-02-11T00:35:52.210610079Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.218162_510.1","section":"bucket.instance","name":"ufpbfn-29:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.29","timestamp":"2021-02-11T00:35:52.218162393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.221895_511.1","section":"bucket","name":"ufpbfn-29","timestamp":"2021-02-11T00:35:52.221895004Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JVPKNH1TNGfzqm12utWovVB","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.232622_512.1","section":"bucket","name":"ufpbfn-29","timestamp":"2021-02-11T00:35:52.232622253Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JVPKNH1TNGfzqm12utWovVB","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.255266_513.1","section":"bucket.instance","name":"ufpbfn-27:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.27","timestamp":"2021-02-11T00:35:52.255266759Z","data":{"read_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":1},"write_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.265160_514.1","section":"bucket.instance","name":"ufpbfn-27:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.27","timestamp":"2021-02-11T00:35:52.265160032Z","data":{"read_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":1},"write_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.315161_515.1","section":"bucket.instance","name":"ufpbfn-28:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.28","timestamp":"2021-02-11T00:35:52.315161773Z","data":{"read_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":1},"write_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.325134_516.1","section":"bucket.instance","name":"ufpbfn-28:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.28","timestamp":"2021-02-11T00:35:52.325134952Z","data":{"read_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":1},"write_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.382688_517.1","section":"bucket.instance","name":"ufpbfn-29:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.29","timestamp":"2021-02-11T00:35:52.382688910Z","data":{"read_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":1},"write_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.393754_518.1","section":"bucket.instance","name":"ufpbfn-29:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.29","timestamp":"2021-02-11T00:35:52.393754381Z","data":{"read_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":1},"write_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.469936_519.1","section":"bucket.instance","name":"ufpbfn-30:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.30","timestamp":"2021-02-11T00:35:52.469936591Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.475466_520.1","section":"bucket.instance","name":"ufpbfn-30:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.30","timestamp":"2021-02-11T00:35:52.475466058Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.477315_521.1","section":"bucket","name":"ufpbfn-30","timestamp":"2021-02-11T00:35:52.477315559Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jjRH-goef-lAu1Abk5YXeix","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.481744_522.1","section":"bucket","name":"ufpbfn-30","timestamp":"2021-02-11T00:35:52.481744078Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jjRH-goef-lAu1Abk5YXeix","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.502453_523.1","section":"bucket.instance","name":"ufpbfn-31:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.31","timestamp":"2021-02-11T00:35:52.502453639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.508136_524.1","section":"bucket.instance","name":"ufpbfn-31:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.31","timestamp":"2021-02-11T00:35:52.508136124Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.510150_525.1","section":"bucket","name":"ufpbfn-31","timestamp":"2021-02-11T00:35:52.510150885Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YkTpeSeHlvxSYw5Rw8Ab4_g","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.515456_526.1","section":"bucket","name":"ufpbfn-31","timestamp":"2021-02-11T00:35:52.515456566Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YkTpeSeHlvxSYw5Rw8Ab4_g","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.561311_527.1","section":"bucket.instance","name":"ufpbfn-32:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.32","timestamp":"2021-02-11T00:35:52.561311410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.567433_528.1","section":"bucket.instance","name":"ufpbfn-32:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.32","timestamp":"2021-02-11T00:35:52.567433316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.569487_529.1","section":"bucket","name":"ufpbfn-32","timestamp":"2021-02-11T00:35:52.569487932Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LHW-hTrz97v-1TYWLUyydCX","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.574147_530.1","section":"bucket","name":"ufpbfn-32","timestamp":"2021-02-11T00:35:52.574147158Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LHW-hTrz97v-1TYWLUyydCX","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.597912_531.1","section":"bucket.instance","name":"ufpbfn-30:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.30","timestamp":"2021-02-11T00:35:52.597912475Z","data":{"read_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":1},"write_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.604469_532.1","section":"bucket.instance","name":"ufpbfn-30:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.30","timestamp":"2021-02-11T00:35:52.604469492Z","data":{"read_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":1},"write_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.656328_533.1","section":"bucket.instance","name":"ufpbfn-31:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.31","timestamp":"2021-02-11T00:35:52.656328101Z","data":{"read_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":1},"write_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.668788_534.1","section":"bucket.instance","name":"ufpbfn-31:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.31","timestamp":"2021-02-11T00:35:52.668788500Z","data":{"read_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":1},"write_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.724450_535.1","section":"bucket.instance","name":"ufpbfn-32:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.32","timestamp":"2021-02-11T00:35:52.724450815Z","data":{"read_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":1},"write_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.732591_536.1","section":"bucket.instance","name":"ufpbfn-32:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.32","timestamp":"2021-02-11T00:35:52.732591666Z","data":{"read_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":1},"write_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.808374_537.1","section":"bucket.instance","name":"ufpbfn-33:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.33","timestamp":"2021-02-11T00:35:52.808374402Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.815859_538.1","section":"bucket.instance","name":"ufpbfn-33:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.33","timestamp":"2021-02-11T00:35:52.815859569Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.821293_539.1","section":"bucket","name":"ufpbfn-33","timestamp":"2021-02-11T00:35:52.821293819Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qyZG5IHPnJ0q7XL8oHBFG_M","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.833861_540.1","section":"bucket","name":"ufpbfn-33","timestamp":"2021-02-11T00:35:52.833861919Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qyZG5IHPnJ0q7XL8oHBFG_M","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.855079_541.1","section":"bucket.instance","name":"ufpbfn-34:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.34","timestamp":"2021-02-11T00:35:52.855079273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.863679_542.1","section":"bucket.instance","name":"ufpbfn-34:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.34","timestamp":"2021-02-11T00:35:52.863679302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.865715_543.1","section":"bucket","name":"ufpbfn-34","timestamp":"2021-02-11T00:35:52.865715392Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eaPmsUe9UpKZde4q81ZEGn-","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.870203_544.1","section":"bucket","name":"ufpbfn-34","timestamp":"2021-02-11T00:35:52.870203091Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eaPmsUe9UpKZde4q81ZEGn-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.911282_545.1","section":"bucket.instance","name":"ufpbfn-35:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.35","timestamp":"2021-02-11T00:35:52.911282623Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.917850_546.1","section":"bucket.instance","name":"ufpbfn-35:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.35","timestamp":"2021-02-11T00:35:52.917850358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.920196_547.1","section":"bucket","name":"ufpbfn-35","timestamp":"2021-02-11T00:35:52.920196626Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2P7hUA6fxJFof13fab6ELfJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.925674_548.1","section":"bucket","name":"ufpbfn-35","timestamp":"2021-02-11T00:35:52.925674365Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2P7hUA6fxJFof13fab6ELfJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.946987_549.1","section":"bucket.instance","name":"ufpbfn-33:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.33","timestamp":"2021-02-11T00:35:52.946987783Z","data":{"read_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":1},"write_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.952944_550.1","section":"bucket.instance","name":"ufpbfn-33:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.33","timestamp":"2021-02-11T00:35:52.952944552Z","data":{"read_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":1},"write_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003754.007637_551.1","section":"bucket.instance","name":"ufpbfn-34:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.34","timestamp":"2021-02-11T00:35:54.007637789Z","data":{"read_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":1},"write_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":2},"status":{"status":"write"}}},{"id":"1_1613003754.017772_552.1","section":"bucket.instance","name":"ufpbfn-34:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.34","timestamp":"2021-02-11T00:35:54.017772278Z","data":{"read_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":1},"write_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003755.079943_553.1","section":"bucket.instance","name":"ufpbfn-35:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.35","timestamp":"2021-02-11T00:35:55.079943426Z","data":{"read_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":1},"write_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":2},"status":{"status":"write"}}},{"id":"1_1613003755.092005_554.1","section":"bucket.instance","name":"ufpbfn-35:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.35","timestamp":"2021-02-11T00:35:55.092005034Z","data":{"read_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":1},"write_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003755.170194_555.1","section":"bucket.instance","name":"ufpbfn-36:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.36","timestamp":"2021-02-11T00:35:55.170194935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6Bhkdo_dUOAdMeAbwoirslC","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.177626_556.1","section":"bucket.instance","name":"ufpbfn-36:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.36","timestamp":"2021-02-11T00:35:55.177626644Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6Bhkdo_dUOAdMeAbwoirslC","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.180328_557.1","section":"bucket","name":"ufpbfn-36","timestamp":"2021-02-11T00:35:55.180328782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yhkND0d6ef1KagNiHqzkC3_","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.186832_558.1","section":"bucket","name":"ufpbfn-36","timestamp":"2021-02-11T00:35:55.186832130Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yhkND0d6ef1KagNiHqzkC3_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.224261_559.1","section":"bucket.instance","name":"ufpbfn-37:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.37","timestamp":"2021-02-11T00:35:55.224261590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j2fBBQD3zN4CmLC7L14TrCt","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.232953_560.1","section":"bucket.instance","name":"ufpbfn-37:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.37","timestamp":"2021-02-11T00:35:55.232953898Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j2fBBQD3zN4CmLC7L14TrCt","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.235184_561.1","section":"bucket","name":"ufpbfn-37","timestamp":"2021-02-11T00:35:55.235184111Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ET2FdSdbJ8ZEOCAPZJ-po8a","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.239904_562.1","section":"bucket","name":"ufpbfn-37","timestamp":"2021-02-11T00:35:55.239904921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ET2FdSdbJ8ZEOCAPZJ-po8a","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.279481_563.1","section":"bucket.instance","name":"ufpbfn-38:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.38","timestamp":"2021-02-11T00:35:55.279481119Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DmS3IetBwqoZFY34Xk3Ohek","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.284473_564.1","section":"bucket.instance","name":"ufpbfn-38:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.38","timestamp":"2021-02-11T00:35:55.284473066Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DmS3IetBwqoZFY34Xk3Ohek","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.286540_565.1","section":"bucket","name":"ufpbfn-38","timestamp":"2021-02-11T00:35:55.286540135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0ru7BE0bzAXx0-Pb22dsKmU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.291866_566.1","section":"bucket","name":"ufpbfn-38","timestamp":"2021-02-11T00:35:55.291866653Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0ru7BE0bzAXx0-Pb22dsKmU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.788416_570.1","section":"bucket.instance","name":"ufpbfn-39:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.39","timestamp":"2021-02-11T00:35:57.788416591Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1vgWQA2CxVJU7vO6qXuqW2w","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.796322_571.1","section":"bucket.instance","name":"ufpbfn-39:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.39","timestamp":"2021-02-11T00:35:57.796322629Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1vgWQA2CxVJU7vO6qXuqW2w","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.798608_572.1","section":"bucket","name":"ufpbfn-39","timestamp":"2021-02-11T00:35:57.798608517Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kSxSKKo8ruuBEqht7yFAiKh","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.804310_573.1","section":"bucket","name":"ufpbfn-39","timestamp":"2021-02-11T00:35:57.804310795Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kSxSKKo8ruuBEqht7yFAiKh","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.834923_574.1","section":"bucket.instance","name":"ufpbfn-40:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.40","timestamp":"2021-02-11T00:35:57.834923842Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jcrlr-QMPmHhG1khCqUK-HL","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.844273_575.1","section":"bucket.instance","name":"ufpbfn-40:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.40","timestamp":"2021-02-11T00:35:57.844273207Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jcrlr-QMPmHhG1khCqUK-HL","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.846725_576.1","section":"bucket","name":"ufpbfn-40","timestamp":"2021-02-11T00:35:57.846725006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-U2oP92YIGzVDnZTkauQDfU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.851372_577.1","section":"bucket","name":"ufpbfn-40","timestamp":"2021-02-11T00:35:57.851372809Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-U2oP92YIGzVDnZTkauQDfU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.893851_578.1","section":"bucket.instance","name":"ufpbfn-41:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.41","timestamp":"2021-02-11T00:35:57.893851494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_F6KHu-saDEKBY8ttlAXOWyy","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.902224_579.1","section":"bucket.instance","name":"ufpbfn-41:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.41","timestamp":"2021-02-11T00:35:57.902224837Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_F6KHu-saDEKBY8ttlAXOWyy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.904331_580.1","section":"bucket","name":"ufpbfn-41","timestamp":"2021-02-11T00:35:57.904331122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Hr9JbpKcJAgiTFrG2I388EO","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.913330_581.1","section":"bucket","name":"ufpbfn-41","timestamp":"2021-02-11T00:35:57.913330476Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Hr9JbpKcJAgiTFrG2I388EO","ver":1},"status":{"status":"complete"}}}] 2021-02-11T00:38:16.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '7f3547bf-73e3-43a7-aa91-648db0f982f6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:16.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '7f3547bf-73e3-43a7-aa91-648db0f982f6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:16.398 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 7f3547bf-73e3-43a7-aa91-648db0f982f6 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:16.437 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:16.438 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:16.691 INFO:teuthology.orchestra.run.smithi095.stdout:[{"id":"1_1613003450.854302_44.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T00:30:50.854302550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8afXW5L0598IonMpxK19QLU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003450.863150_45.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-02-11T00:30:50.863150983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8afXW5L0598IonMpxK19QLU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003573.921741_48.1","section":"bucket.instance","name":"ufpbfn-1:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.1","timestamp":"2021-02-11T00:32:53.921741821Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Skjph_QFacdJPxfoZZsFlwv","ver":1},"status":{"status":"write"}}},{"id":"1_1613003573.938336_49.1","section":"bucket.instance","name":"ufpbfn-1:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.1","timestamp":"2021-02-11T00:32:53.938336161Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Skjph_QFacdJPxfoZZsFlwv","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003573.939868_50.1","section":"bucket","name":"ufpbfn-1","timestamp":"2021-02-11T00:32:53.939868855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KqGKnzK6Tf8QvJEeDxt1_WH","ver":1},"status":{"status":"write"}}},{"id":"1_1613003573.944120_51.1","section":"bucket","name":"ufpbfn-1","timestamp":"2021-02-11T00:32:53.944120215Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KqGKnzK6Tf8QvJEeDxt1_WH","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003573.965387_52.1","section":"bucket.instance","name":"ufpbfn-2:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.2","timestamp":"2021-02-11T00:32:53.965387242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_opo5r9iAQSy3iT3OqymSbUR","ver":1},"status":{"status":"write"}}},{"id":"1_1613003573.970287_53.1","section":"bucket.instance","name":"ufpbfn-2:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.2","timestamp":"2021-02-11T00:32:53.970287398Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_opo5r9iAQSy3iT3OqymSbUR","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003573.971912_54.1","section":"bucket","name":"ufpbfn-2","timestamp":"2021-02-11T00:32:53.971912556Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I8Y2RjNUm2--MMKap47mPvi","ver":1},"status":{"status":"write"}}},{"id":"1_1613003573.976646_55.1","section":"bucket","name":"ufpbfn-2","timestamp":"2021-02-11T00:32:53.976646634Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I8Y2RjNUm2--MMKap47mPvi","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003577.130440_56.1","section":"bucket.instance","name":"ufpbfn-3:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.3","timestamp":"2021-02-11T00:32:57.130440180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__JIYMfX43-rh93xo8qCfW3W","ver":1},"status":{"status":"write"}}},{"id":"1_1613003577.138907_57.1","section":"bucket.instance","name":"ufpbfn-3:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.3","timestamp":"2021-02-11T00:32:57.138907718Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__JIYMfX43-rh93xo8qCfW3W","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003577.141172_58.1","section":"bucket","name":"ufpbfn-3","timestamp":"2021-02-11T00:32:57.141172704Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZD69Kxj7MfN4ykEqn2AmMHP","ver":1},"status":{"status":"write"}}},{"id":"1_1613003577.148691_59.1","section":"bucket","name":"ufpbfn-3","timestamp":"2021-02-11T00:32:57.148691001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZD69Kxj7MfN4ykEqn2AmMHP","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.351288_60.1","section":"bucket.instance","name":"ufpbfn-4:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.4","timestamp":"2021-02-11T00:32:58.351288084Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7b_KGvalODv_9Uf9Kk5icn8","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.356570_61.1","section":"bucket.instance","name":"ufpbfn-4:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.4","timestamp":"2021-02-11T00:32:58.356570584Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7b_KGvalODv_9Uf9Kk5icn8","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.358632_62.1","section":"bucket","name":"ufpbfn-4","timestamp":"2021-02-11T00:32:58.358632530Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jr7oDJUzCTW6gHPoeufR7jm","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.363794_63.1","section":"bucket","name":"ufpbfn-4","timestamp":"2021-02-11T00:32:58.363794123Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jr7oDJUzCTW6gHPoeufR7jm","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.382871_64.1","section":"bucket.instance","name":"ufpbfn-5:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.5","timestamp":"2021-02-11T00:32:58.382871708Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a0Y62ZJ3YMY_wT-paa_QHFa","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.388437_65.1","section":"bucket.instance","name":"ufpbfn-5:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.5","timestamp":"2021-02-11T00:32:58.388437173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a0Y62ZJ3YMY_wT-paa_QHFa","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.390530_66.1","section":"bucket","name":"ufpbfn-5","timestamp":"2021-02-11T00:32:58.390530916Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZMPDqfQwJlwt4fOPIl0M8aR","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.395610_67.1","section":"bucket","name":"ufpbfn-5","timestamp":"2021-02-11T00:32:58.395610305Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZMPDqfQwJlwt4fOPIl0M8aR","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.436947_68.1","section":"bucket.instance","name":"ufpbfn-6:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.6","timestamp":"2021-02-11T00:32:58.436947070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iVj6C98NL7g1XpDBkqjdwxs","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.442139_69.1","section":"bucket.instance","name":"ufpbfn-6:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.6","timestamp":"2021-02-11T00:32:58.442139964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iVj6C98NL7g1XpDBkqjdwxs","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003578.443881_70.1","section":"bucket","name":"ufpbfn-6","timestamp":"2021-02-11T00:32:58.443881886Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gm9dBc_bz3ThL3EzllaIRon","ver":1},"status":{"status":"write"}}},{"id":"1_1613003578.448531_71.1","section":"bucket","name":"ufpbfn-6","timestamp":"2021-02-11T00:32:58.448531161Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gm9dBc_bz3ThL3EzllaIRon","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.032175_72.1","section":"bucket.instance","name":"ufpbfn-7:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.7","timestamp":"2021-02-11T00:33:01.032175874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6WrCP2fZ4q43uXb3VOBaP-X","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.037898_73.1","section":"bucket.instance","name":"ufpbfn-7:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.7","timestamp":"2021-02-11T00:33:01.037898478Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6WrCP2fZ4q43uXb3VOBaP-X","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.040379_74.1","section":"bucket","name":"ufpbfn-7","timestamp":"2021-02-11T00:33:01.040379029Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.046480_75.1","section":"bucket","name":"ufpbfn-7","timestamp":"2021-02-11T00:33:01.046480658Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.066133_76.1","section":"bucket.instance","name":"ufpbfn-8:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.8","timestamp":"2021-02-11T00:33:01.066133971Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ImgVKlW5BV_4yXJ5eaN1blE","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.071786_77.1","section":"bucket.instance","name":"ufpbfn-8:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.8","timestamp":"2021-02-11T00:33:01.071786138Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ImgVKlW5BV_4yXJ5eaN1blE","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.073784_78.1","section":"bucket","name":"ufpbfn-8","timestamp":"2021-02-11T00:33:01.073784230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.078439_79.1","section":"bucket","name":"ufpbfn-8","timestamp":"2021-02-11T00:33:01.078439230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.118285_80.1","section":"bucket.instance","name":"ufpbfn-9:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.9","timestamp":"2021-02-11T00:33:01.118285107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4WCKMP2jWFDUs3dtJoGmuxJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.123404_81.1","section":"bucket.instance","name":"ufpbfn-9:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.9","timestamp":"2021-02-11T00:33:01.123404230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4WCKMP2jWFDUs3dtJoGmuxJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003581.125274_82.1","section":"bucket","name":"ufpbfn-9","timestamp":"2021-02-11T00:33:01.125274841Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003581.130865_83.1","section":"bucket","name":"ufpbfn-9","timestamp":"2021-02-11T00:33:01.130865699Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003582.332946_84.1","section":"bucket","name":"ufpbfn-7","timestamp":"2021-02-11T00:33:02.332946911Z","data":{"read_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":1},"write_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":2},"status":{"status":"remove"}}},{"id":"1_1613003582.340020_85.1","section":"bucket","name":"ufpbfn-7","timestamp":"2021-02-11T00:33:02.340020754Z","data":{"read_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":1},"write_version":{"tag":"_yFDM0HS7rc9WeLhUdZGqDtD","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003582.420876_86.1","section":"bucket","name":"ufpbfn-8","timestamp":"2021-02-11T00:33:02.420876072Z","data":{"read_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":1},"write_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":2},"status":{"status":"remove"}}},{"id":"1_1613003582.427710_87.1","section":"bucket","name":"ufpbfn-8","timestamp":"2021-02-11T00:33:02.427710584Z","data":{"read_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":1},"write_version":{"tag":"_hnnnC1JnXh7NTf80bUHN78u","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003582.554110_88.1","section":"bucket","name":"ufpbfn-9","timestamp":"2021-02-11T00:33:02.554110401Z","data":{"read_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":1},"write_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":2},"status":{"status":"remove"}}},{"id":"1_1613003582.559316_89.1","section":"bucket","name":"ufpbfn-9","timestamp":"2021-02-11T00:33:02.559316632Z","data":{"read_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":1},"write_version":{"tag":"_CINlfwYwGzodJJj4suPMefU","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003583.730202_48.1","section":"bucket.instance","name":"ufpbfn-10:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.10","timestamp":"2021-02-11T00:33:03.730202256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2eGqXaG3AOEn7phUs2DcOIc","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.736098_58.1","section":"bucket.instance","name":"ufpbfn-10:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.10","timestamp":"2021-02-11T00:33:03.736098882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2eGqXaG3AOEn7phUs2DcOIc","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.738359_59.1","section":"bucket","name":"ufpbfn-10","timestamp":"2021-02-11T00:33:03.738359979Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tDxlkqHxmS7PTDEab2H7hMT","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.744305_60.1","section":"bucket","name":"ufpbfn-10","timestamp":"2021-02-11T00:33:03.744305538Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tDxlkqHxmS7PTDEab2H7hMT","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.767444_61.1","section":"bucket.instance","name":"ufpbfn-11:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.11","timestamp":"2021-02-11T00:33:03.767444315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wkX6SG6rOg3K4s40k0F4qy5","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.773359_62.1","section":"bucket.instance","name":"ufpbfn-11:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.11","timestamp":"2021-02-11T00:33:03.773359088Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wkX6SG6rOg3K4s40k0F4qy5","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.775379_63.1","section":"bucket","name":"ufpbfn-11","timestamp":"2021-02-11T00:33:03.775379051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U4YSB-MjK3Lwa5aeb8ZJsWC","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.780715_64.1","section":"bucket","name":"ufpbfn-11","timestamp":"2021-02-11T00:33:03.780715854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U4YSB-MjK3Lwa5aeb8ZJsWC","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.825241_65.1","section":"bucket.instance","name":"ufpbfn-12:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.12","timestamp":"2021-02-11T00:33:03.825241462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jq-H07HPoS-gctsMOEeF9RO","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.830032_66.1","section":"bucket.instance","name":"ufpbfn-12:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.12","timestamp":"2021-02-11T00:33:03.830032565Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jq-H07HPoS-gctsMOEeF9RO","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003583.832053_67.1","section":"bucket","name":"ufpbfn-12","timestamp":"2021-02-11T00:33:03.832053315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__fBfLcBSoKwIw5StU9Nvh6L","ver":1},"status":{"status":"write"}}},{"id":"1_1613003583.836924_68.1","section":"bucket","name":"ufpbfn-12","timestamp":"2021-02-11T00:33:03.836924032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__fBfLcBSoKwIw5StU9Nvh6L","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.659565_73.1","section":"bucket.instance","name":"ufpbfn-13:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.13","timestamp":"2021-02-11T00:33:17.659565432Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f8SIjk338xKgtCDgHkGKqff","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.665073_76.1","section":"bucket.instance","name":"ufpbfn-13:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.13","timestamp":"2021-02-11T00:33:17.665073710Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f8SIjk338xKgtCDgHkGKqff","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.666958_77.1","section":"bucket","name":"ufpbfn-13","timestamp":"2021-02-11T00:33:17.666958458Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0NQ3xjtUNQUmGx9jbXsmf2Z","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.671944_78.1","section":"bucket","name":"ufpbfn-13","timestamp":"2021-02-11T00:33:17.671944640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0NQ3xjtUNQUmGx9jbXsmf2Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.692682_79.1","section":"bucket.instance","name":"ufpbfn-14:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.14","timestamp":"2021-02-11T00:33:17.692682553Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L43RwNuj0ddF9ifaSHgIKye","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.699242_80.1","section":"bucket.instance","name":"ufpbfn-14:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.14","timestamp":"2021-02-11T00:33:17.699242273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_L43RwNuj0ddF9ifaSHgIKye","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.701224_81.1","section":"bucket","name":"ufpbfn-14","timestamp":"2021-02-11T00:33:17.701224321Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MRG403-5B5xPKMkhFWKEsyk","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.706794_82.1","section":"bucket","name":"ufpbfn-14","timestamp":"2021-02-11T00:33:17.706794690Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MRG403-5B5xPKMkhFWKEsyk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.754649_83.1","section":"bucket.instance","name":"ufpbfn-15:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.15","timestamp":"2021-02-11T00:33:17.754649827Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KrcZ_amVznZV1n91M3anyap","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.765027_84.1","section":"bucket.instance","name":"ufpbfn-15:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.15","timestamp":"2021-02-11T00:33:17.765027652Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KrcZ_amVznZV1n91M3anyap","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003597.767014_85.1","section":"bucket","name":"ufpbfn-15","timestamp":"2021-02-11T00:33:17.767014291Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EgRyHS2CuJZW4_js4EagF7N","ver":1},"status":{"status":"write"}}},{"id":"1_1613003597.771457_86.1","section":"bucket","name":"ufpbfn-15","timestamp":"2021-02-11T00:33:17.771457040Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EgRyHS2CuJZW4_js4EagF7N","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.388153_87.1","section":"bucket.instance","name":"ufpbfn-16:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.16","timestamp":"2021-02-11T00:33:26.388153899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.394315_93.1","section":"bucket.instance","name":"ufpbfn-16:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.16","timestamp":"2021-02-11T00:33:26.394315586Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.396589_94.1","section":"bucket","name":"ufpbfn-16","timestamp":"2021-02-11T00:33:26.396589995Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c6TEuYH8lTtdHiiKB2PN7B3","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.401860_95.1","section":"bucket","name":"ufpbfn-16","timestamp":"2021-02-11T00:33:26.401860934Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c6TEuYH8lTtdHiiKB2PN7B3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.426046_96.1","section":"bucket.instance","name":"ufpbfn-17:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.17","timestamp":"2021-02-11T00:33:26.426046415Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.431552_97.1","section":"bucket.instance","name":"ufpbfn-17:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.17","timestamp":"2021-02-11T00:33:26.431552117Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.433354_98.1","section":"bucket","name":"ufpbfn-17","timestamp":"2021-02-11T00:33:26.433354162Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6elO0Q-rH28Qr3aVT2qAwx6","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.438176_99.1","section":"bucket","name":"ufpbfn-17","timestamp":"2021-02-11T00:33:26.438176837Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6elO0Q-rH28Qr3aVT2qAwx6","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.480439_101.1","section":"bucket.instance","name":"ufpbfn-18:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.18","timestamp":"2021-02-11T00:33:26.480439651Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.486235_102.1","section":"bucket.instance","name":"ufpbfn-18:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.18","timestamp":"2021-02-11T00:33:26.486235046Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.488060_103.1","section":"bucket","name":"ufpbfn-18","timestamp":"2021-02-11T00:33:26.488060035Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s4Ba4emrAs-SlpdvF9nVDgT","ver":1},"status":{"status":"write"}}},{"id":"1_1613003606.492951_104.1","section":"bucket","name":"ufpbfn-18","timestamp":"2021-02-11T00:33:26.492951271Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s4Ba4emrAs-SlpdvF9nVDgT","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003606.515036_105.1","section":"bucket.instance","name":"ufpbfn-16:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.16","timestamp":"2021-02-11T00:33:26.515036772Z","data":{"read_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":1},"write_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":2},"status":{"status":"write"}}},{"id":"1_1613003606.520310_106.1","section":"bucket.instance","name":"ufpbfn-16:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.16","timestamp":"2021-02-11T00:33:26.520310165Z","data":{"read_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":1},"write_version":{"tag":"_3lLOgzWUcvQ0yzQkT9Ug4g1","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003606.525055_107.1","section":"bucket.instance","name":"ufpbfn-17:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.17","timestamp":"2021-02-11T00:33:26.525055588Z","data":{"read_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":1},"write_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":2},"status":{"status":"write"}}},{"id":"1_1613003606.531303_108.1","section":"bucket.instance","name":"ufpbfn-17:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.17","timestamp":"2021-02-11T00:33:26.531303087Z","data":{"read_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":1},"write_version":{"tag":"_s8h2hIfiRoINIlTV8my2Lmk","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003606.542749_110.1","section":"bucket.instance","name":"ufpbfn-18:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.18","timestamp":"2021-02-11T00:33:26.542749975Z","data":{"read_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":1},"write_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":2},"status":{"status":"write"}}},{"id":"1_1613003606.555218_111.1","section":"bucket.instance","name":"ufpbfn-18:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.18","timestamp":"2021-02-11T00:33:26.555218578Z","data":{"read_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":1},"write_version":{"tag":"_2fzJn4NC2r7vhMrqLO3lx34","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003663.038735_296.1","section":"bucket.instance","name":"ufpbfn-19:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.19","timestamp":"2021-02-11T00:34:23.038735977Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":1},"status":{"status":"write"}}},{"id":"1_1613003663.044326_299.1","section":"bucket.instance","name":"ufpbfn-19:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.19","timestamp":"2021-02-11T00:34:23.044326194Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003663.046352_300.1","section":"bucket","name":"ufpbfn-19","timestamp":"2021-02-11T00:34:23.046352692Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_quU3_y1npE6yjsgMfaiPzD-","ver":1},"status":{"status":"write"}}},{"id":"1_1613003663.051849_301.1","section":"bucket","name":"ufpbfn-19","timestamp":"2021-02-11T00:34:23.051849389Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_quU3_y1npE6yjsgMfaiPzD-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003663.058645_302.1","section":"bucket.instance","name":"ufpbfn-19:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.19","timestamp":"2021-02-11T00:34:23.058645404Z","data":{"read_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":1},"write_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":2},"status":{"status":"write"}}},{"id":"1_1613003663.066081_303.1","section":"bucket.instance","name":"ufpbfn-19:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.19","timestamp":"2021-02-11T00:34:23.066081205Z","data":{"read_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":1},"write_version":{"tag":"_YWNNW91r_lpjMx5kP1oNqWu","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003710.698748_304.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:10.698748192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":1},"status":{"status":"write"}}},{"id":"1_1613003710.704651_318.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:10.704651509Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003710.706704_319.1","section":"bucket","name":"ufpbfn-20","timestamp":"2021-02-11T00:35:10.706704347Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OoZxlXdV7j7UCHfz9fjr3nn","ver":1},"status":{"status":"write"}}},{"id":"1_1613003710.715573_320.1","section":"bucket","name":"ufpbfn-20","timestamp":"2021-02-11T00:35:10.715573590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OoZxlXdV7j7UCHfz9fjr3nn","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003719.828914_353.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:19.828914598Z","data":{"read_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":1},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":2},"status":{"status":"write"}}},{"id":"1_1613003719.838940_354.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:19.838940378Z","data":{"read_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":1},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003726.262512_399.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:26.262512788Z","data":{"read_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":2},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":3},"status":{"status":"write"}}},{"id":"1_1613003726.272815_400.1","section":"bucket.instance","name":"ufpbfn-20:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.20","timestamp":"2021-02-11T00:35:26.272815115Z","data":{"read_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":2},"write_version":{"tag":"_r8PtETvs2_zIh0ekESi8tC_","ver":3},"status":{"status":"complete"}}},{"id":"1_1613003730.762664_445.1","section":"bucket.instance","name":"ufpbfn-21:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.21","timestamp":"2021-02-11T00:35:30.762664019Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.772882_446.1","section":"bucket.instance","name":"ufpbfn-21:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.21","timestamp":"2021-02-11T00:35:30.772882870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.775134_447.1","section":"bucket","name":"ufpbfn-21","timestamp":"2021-02-11T00:35:30.775134980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k5uY3Gk97MZZN-lG6Hb8Fye","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.783979_448.1","section":"bucket","name":"ufpbfn-21","timestamp":"2021-02-11T00:35:30.783979099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k5uY3Gk97MZZN-lG6Hb8Fye","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.811133_449.1","section":"bucket.instance","name":"ufpbfn-22:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.22","timestamp":"2021-02-11T00:35:30.811133053Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.816622_450.1","section":"bucket.instance","name":"ufpbfn-22:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.22","timestamp":"2021-02-11T00:35:30.816622426Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.818531_451.1","section":"bucket","name":"ufpbfn-22","timestamp":"2021-02-11T00:35:30.818531149Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FC7QfdFkMCENzYwmhnYnLvS","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.823109_452.1","section":"bucket","name":"ufpbfn-22","timestamp":"2021-02-11T00:35:30.823109099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FC7QfdFkMCENzYwmhnYnLvS","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.866148_453.1","section":"bucket.instance","name":"ufpbfn-23:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.23","timestamp":"2021-02-11T00:35:30.866148114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.871906_454.1","section":"bucket.instance","name":"ufpbfn-23:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.23","timestamp":"2021-02-11T00:35:30.871906109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.875362_455.1","section":"bucket","name":"ufpbfn-23","timestamp":"2021-02-11T00:35:30.875362846Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JXU6nBLcF_sOtZUty9mXONE","ver":1},"status":{"status":"write"}}},{"id":"1_1613003730.880899_456.1","section":"bucket","name":"ufpbfn-23","timestamp":"2021-02-11T00:35:30.880899190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JXU6nBLcF_sOtZUty9mXONE","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003730.906538_457.1","section":"bucket.instance","name":"ufpbfn-21:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.21","timestamp":"2021-02-11T00:35:30.906538677Z","data":{"read_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":1},"write_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":2},"status":{"status":"write"}}},{"id":"1_1613003730.912385_458.1","section":"bucket.instance","name":"ufpbfn-21:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.21","timestamp":"2021-02-11T00:35:30.912385378Z","data":{"read_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":1},"write_version":{"tag":"_-XqV79l28s2EJ6KFpPoh9ZJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003730.916595_459.1","section":"bucket.instance","name":"ufpbfn-22:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.22","timestamp":"2021-02-11T00:35:30.916595556Z","data":{"read_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":1},"write_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":2},"status":{"status":"write"}}},{"id":"1_1613003730.921350_460.1","section":"bucket.instance","name":"ufpbfn-22:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.22","timestamp":"2021-02-11T00:35:30.921350521Z","data":{"read_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":1},"write_version":{"tag":"_dNpiRSUjIiFw50embB9mK0U","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003730.930587_461.1","section":"bucket.instance","name":"ufpbfn-23:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.23","timestamp":"2021-02-11T00:35:30.930587206Z","data":{"read_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":1},"write_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":2},"status":{"status":"write"}}},{"id":"1_1613003730.936764_462.1","section":"bucket.instance","name":"ufpbfn-23:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.23","timestamp":"2021-02-11T00:35:30.936764259Z","data":{"read_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":1},"write_version":{"tag":"_MDM2uHwz1mO-0h86Jj7Ld8H","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003742.361724_463.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.361724903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.367316_471.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.367316098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.369891_472.1","section":"bucket","name":"ufpbfn-24","timestamp":"2021-02-11T00:35:42.369891133Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3K3JZc4KLLW9-gE3essIbqt","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.375942_473.1","section":"bucket","name":"ufpbfn-24","timestamp":"2021-02-11T00:35:42.375942599Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3K3JZc4KLLW9-gE3essIbqt","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.404482_474.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.404482219Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.410336_475.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.410336039Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.412835_476.1","section":"bucket","name":"ufpbfn-25","timestamp":"2021-02-11T00:35:42.412835404Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a-6lD0m5KM_3p3bHNRp9pQd","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.417918_477.1","section":"bucket","name":"ufpbfn-25","timestamp":"2021-02-11T00:35:42.417918083Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a-6lD0m5KM_3p3bHNRp9pQd","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.451495_478.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.451495596Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.458335_479.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.458335827Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.463725_480.1","section":"bucket","name":"ufpbfn-26","timestamp":"2021-02-11T00:35:42.463725770Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_weJpVYgvsyih4Mc-CA_rT6D","ver":1},"status":{"status":"write"}}},{"id":"1_1613003742.471367_481.1","section":"bucket","name":"ufpbfn-26","timestamp":"2021-02-11T00:35:42.471367127Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_weJpVYgvsyih4Mc-CA_rT6D","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003742.492942_482.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.492942914Z","data":{"read_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":1},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":2},"status":{"status":"write"}}},{"id":"1_1613003742.498602_483.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.498602849Z","data":{"read_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":1},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003742.502980_484.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.502980249Z","data":{"read_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":2},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":3},"status":{"status":"write"}}},{"id":"1_1613003742.509352_485.1","section":"bucket.instance","name":"ufpbfn-24:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.24","timestamp":"2021-02-11T00:35:42.509352182Z","data":{"read_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":2},"write_version":{"tag":"_VbeTMlA12nzNOta9AUUDSif","ver":3},"status":{"status":"complete"}}},{"id":"1_1613003742.515116_486.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.515116187Z","data":{"read_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":1},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":2},"status":{"status":"write"}}},{"id":"1_1613003742.522854_487.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.522854749Z","data":{"read_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":1},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003742.535194_488.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.535194409Z","data":{"read_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":2},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":3},"status":{"status":"write"}}},{"id":"1_1613003742.544167_489.1","section":"bucket.instance","name":"ufpbfn-25:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.25","timestamp":"2021-02-11T00:35:42.544167180Z","data":{"read_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":2},"write_version":{"tag":"_nhHmKF5D4OI7fDXQx0K35xO","ver":3},"status":{"status":"complete"}}},{"id":"1_1613003742.556349_490.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.556349382Z","data":{"read_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":1},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":2},"status":{"status":"write"}}},{"id":"1_1613003742.566275_491.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.566275024Z","data":{"read_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":1},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003742.578851_492.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.578851542Z","data":{"read_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":2},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":3},"status":{"status":"write"}}},{"id":"1_1613003742.584739_493.1","section":"bucket.instance","name":"ufpbfn-26:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.26","timestamp":"2021-02-11T00:35:42.584739159Z","data":{"read_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":2},"write_version":{"tag":"_Kii3KCf2Sm09OS2ojpFcgQy","ver":3},"status":{"status":"complete"}}},{"id":"1_1613003752.114576_494.1","section":"bucket.instance","name":"ufpbfn-27:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.27","timestamp":"2021-02-11T00:35:52.114576760Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.120282_502.1","section":"bucket.instance","name":"ufpbfn-27:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.27","timestamp":"2021-02-11T00:35:52.120282248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.122353_503.1","section":"bucket","name":"ufpbfn-27","timestamp":"2021-02-11T00:35:52.122353726Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wtfPlNxQltZmQNvUExRJKda","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.127478_504.1","section":"bucket","name":"ufpbfn-27","timestamp":"2021-02-11T00:35:52.127478495Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wtfPlNxQltZmQNvUExRJKda","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.150327_505.1","section":"bucket.instance","name":"ufpbfn-28:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.28","timestamp":"2021-02-11T00:35:52.150327964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.156621_506.1","section":"bucket.instance","name":"ufpbfn-28:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.28","timestamp":"2021-02-11T00:35:52.156621989Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.158759_507.1","section":"bucket","name":"ufpbfn-28","timestamp":"2021-02-11T00:35:52.158759219Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yn9gTjGHQR-RBwC8BZeqo8y","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.164363_508.1","section":"bucket","name":"ufpbfn-28","timestamp":"2021-02-11T00:35:52.164363564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Yn9gTjGHQR-RBwC8BZeqo8y","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.210610_509.1","section":"bucket.instance","name":"ufpbfn-29:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.29","timestamp":"2021-02-11T00:35:52.210610079Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.218162_510.1","section":"bucket.instance","name":"ufpbfn-29:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.29","timestamp":"2021-02-11T00:35:52.218162393Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.221895_511.1","section":"bucket","name":"ufpbfn-29","timestamp":"2021-02-11T00:35:52.221895004Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JVPKNH1TNGfzqm12utWovVB","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.232622_512.1","section":"bucket","name":"ufpbfn-29","timestamp":"2021-02-11T00:35:52.232622253Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JVPKNH1TNGfzqm12utWovVB","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.255266_513.1","section":"bucket.instance","name":"ufpbfn-27:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.27","timestamp":"2021-02-11T00:35:52.255266759Z","data":{"read_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":1},"write_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.265160_514.1","section":"bucket.instance","name":"ufpbfn-27:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.27","timestamp":"2021-02-11T00:35:52.265160032Z","data":{"read_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":1},"write_version":{"tag":"_kJdGpP2WE1xx_Z7HRoqbtqo","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.315161_515.1","section":"bucket.instance","name":"ufpbfn-28:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.28","timestamp":"2021-02-11T00:35:52.315161773Z","data":{"read_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":1},"write_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.325134_516.1","section":"bucket.instance","name":"ufpbfn-28:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.28","timestamp":"2021-02-11T00:35:52.325134952Z","data":{"read_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":1},"write_version":{"tag":"_nwZtA3jI8F9yaXxhPn6ES4p","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.382688_517.1","section":"bucket.instance","name":"ufpbfn-29:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.29","timestamp":"2021-02-11T00:35:52.382688910Z","data":{"read_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":1},"write_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.393754_518.1","section":"bucket.instance","name":"ufpbfn-29:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.29","timestamp":"2021-02-11T00:35:52.393754381Z","data":{"read_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":1},"write_version":{"tag":"_5fD03IyRM9Pg5FdNVTR68bw","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.469936_519.1","section":"bucket.instance","name":"ufpbfn-30:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.30","timestamp":"2021-02-11T00:35:52.469936591Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.475466_520.1","section":"bucket.instance","name":"ufpbfn-30:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.30","timestamp":"2021-02-11T00:35:52.475466058Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.477315_521.1","section":"bucket","name":"ufpbfn-30","timestamp":"2021-02-11T00:35:52.477315559Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jjRH-goef-lAu1Abk5YXeix","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.481744_522.1","section":"bucket","name":"ufpbfn-30","timestamp":"2021-02-11T00:35:52.481744078Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jjRH-goef-lAu1Abk5YXeix","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.502453_523.1","section":"bucket.instance","name":"ufpbfn-31:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.31","timestamp":"2021-02-11T00:35:52.502453639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.508136_524.1","section":"bucket.instance","name":"ufpbfn-31:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.31","timestamp":"2021-02-11T00:35:52.508136124Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.510150_525.1","section":"bucket","name":"ufpbfn-31","timestamp":"2021-02-11T00:35:52.510150885Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YkTpeSeHlvxSYw5Rw8Ab4_g","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.515456_526.1","section":"bucket","name":"ufpbfn-31","timestamp":"2021-02-11T00:35:52.515456566Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YkTpeSeHlvxSYw5Rw8Ab4_g","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.561311_527.1","section":"bucket.instance","name":"ufpbfn-32:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.32","timestamp":"2021-02-11T00:35:52.561311410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.567433_528.1","section":"bucket.instance","name":"ufpbfn-32:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.32","timestamp":"2021-02-11T00:35:52.567433316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.569487_529.1","section":"bucket","name":"ufpbfn-32","timestamp":"2021-02-11T00:35:52.569487932Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LHW-hTrz97v-1TYWLUyydCX","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.574147_530.1","section":"bucket","name":"ufpbfn-32","timestamp":"2021-02-11T00:35:52.574147158Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LHW-hTrz97v-1TYWLUyydCX","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.597912_531.1","section":"bucket.instance","name":"ufpbfn-30:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.30","timestamp":"2021-02-11T00:35:52.597912475Z","data":{"read_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":1},"write_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.604469_532.1","section":"bucket.instance","name":"ufpbfn-30:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.30","timestamp":"2021-02-11T00:35:52.604469492Z","data":{"read_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":1},"write_version":{"tag":"_7zA6rMVJMgXVw77jzrfxONU","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.656328_533.1","section":"bucket.instance","name":"ufpbfn-31:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.31","timestamp":"2021-02-11T00:35:52.656328101Z","data":{"read_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":1},"write_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.668788_534.1","section":"bucket.instance","name":"ufpbfn-31:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.31","timestamp":"2021-02-11T00:35:52.668788500Z","data":{"read_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":1},"write_version":{"tag":"_otzUZp21IfdF9-oQHcYDdB7","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.724450_535.1","section":"bucket.instance","name":"ufpbfn-32:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.32","timestamp":"2021-02-11T00:35:52.724450815Z","data":{"read_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":1},"write_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.732591_536.1","section":"bucket.instance","name":"ufpbfn-32:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.32","timestamp":"2021-02-11T00:35:52.732591666Z","data":{"read_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":1},"write_version":{"tag":"_5eas2twtJBz3CcNx73R9SDf","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003752.808374_537.1","section":"bucket.instance","name":"ufpbfn-33:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.33","timestamp":"2021-02-11T00:35:52.808374402Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.815859_538.1","section":"bucket.instance","name":"ufpbfn-33:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.33","timestamp":"2021-02-11T00:35:52.815859569Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.821293_539.1","section":"bucket","name":"ufpbfn-33","timestamp":"2021-02-11T00:35:52.821293819Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qyZG5IHPnJ0q7XL8oHBFG_M","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.833861_540.1","section":"bucket","name":"ufpbfn-33","timestamp":"2021-02-11T00:35:52.833861919Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qyZG5IHPnJ0q7XL8oHBFG_M","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.855079_541.1","section":"bucket.instance","name":"ufpbfn-34:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.34","timestamp":"2021-02-11T00:35:52.855079273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.863679_542.1","section":"bucket.instance","name":"ufpbfn-34:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.34","timestamp":"2021-02-11T00:35:52.863679302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.865715_543.1","section":"bucket","name":"ufpbfn-34","timestamp":"2021-02-11T00:35:52.865715392Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eaPmsUe9UpKZde4q81ZEGn-","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.870203_544.1","section":"bucket","name":"ufpbfn-34","timestamp":"2021-02-11T00:35:52.870203091Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eaPmsUe9UpKZde4q81ZEGn-","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.911282_545.1","section":"bucket.instance","name":"ufpbfn-35:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.35","timestamp":"2021-02-11T00:35:52.911282623Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.917850_546.1","section":"bucket.instance","name":"ufpbfn-35:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.35","timestamp":"2021-02-11T00:35:52.917850358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.920196_547.1","section":"bucket","name":"ufpbfn-35","timestamp":"2021-02-11T00:35:52.920196626Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2P7hUA6fxJFof13fab6ELfJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613003752.925674_548.1","section":"bucket","name":"ufpbfn-35","timestamp":"2021-02-11T00:35:52.925674365Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2P7hUA6fxJFof13fab6ELfJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003752.946987_549.1","section":"bucket.instance","name":"ufpbfn-33:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.33","timestamp":"2021-02-11T00:35:52.946987783Z","data":{"read_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":1},"write_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":2},"status":{"status":"write"}}},{"id":"1_1613003752.952944_550.1","section":"bucket.instance","name":"ufpbfn-33:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.33","timestamp":"2021-02-11T00:35:52.952944552Z","data":{"read_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":1},"write_version":{"tag":"_GHio2VTeXwemgFxUZS6fihP","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003754.007637_551.1","section":"bucket.instance","name":"ufpbfn-34:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.34","timestamp":"2021-02-11T00:35:54.007637789Z","data":{"read_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":1},"write_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":2},"status":{"status":"write"}}},{"id":"1_1613003754.017772_552.1","section":"bucket.instance","name":"ufpbfn-34:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.34","timestamp":"2021-02-11T00:35:54.017772278Z","data":{"read_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":1},"write_version":{"tag":"_ZwvkJF8fl-zfSGmErPHIhz3","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003755.079943_553.1","section":"bucket.instance","name":"ufpbfn-35:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.35","timestamp":"2021-02-11T00:35:55.079943426Z","data":{"read_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":1},"write_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":2},"status":{"status":"write"}}},{"id":"1_1613003755.092005_554.1","section":"bucket.instance","name":"ufpbfn-35:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.35","timestamp":"2021-02-11T00:35:55.092005034Z","data":{"read_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":1},"write_version":{"tag":"_--kQTfR-irZY_0XhU_ZHUDm","ver":2},"status":{"status":"complete"}}},{"id":"1_1613003755.170194_555.1","section":"bucket.instance","name":"ufpbfn-36:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.36","timestamp":"2021-02-11T00:35:55.170194935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6Bhkdo_dUOAdMeAbwoirslC","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.177626_556.1","section":"bucket.instance","name":"ufpbfn-36:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.36","timestamp":"2021-02-11T00:35:55.177626644Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6Bhkdo_dUOAdMeAbwoirslC","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.180328_557.1","section":"bucket","name":"ufpbfn-36","timestamp":"2021-02-11T00:35:55.180328782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yhkND0d6ef1KagNiHqzkC3_","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.186832_558.1","section":"bucket","name":"ufpbfn-36","timestamp":"2021-02-11T00:35:55.186832130Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yhkND0d6ef1KagNiHqzkC3_","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.224261_559.1","section":"bucket.instance","name":"ufpbfn-37:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.37","timestamp":"2021-02-11T00:35:55.224261590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j2fBBQD3zN4CmLC7L14TrCt","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.232953_560.1","section":"bucket.instance","name":"ufpbfn-37:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.37","timestamp":"2021-02-11T00:35:55.232953898Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_j2fBBQD3zN4CmLC7L14TrCt","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.235184_561.1","section":"bucket","name":"ufpbfn-37","timestamp":"2021-02-11T00:35:55.235184111Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ET2FdSdbJ8ZEOCAPZJ-po8a","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.239904_562.1","section":"bucket","name":"ufpbfn-37","timestamp":"2021-02-11T00:35:55.239904921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ET2FdSdbJ8ZEOCAPZJ-po8a","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.279481_563.1","section":"bucket.instance","name":"ufpbfn-38:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.38","timestamp":"2021-02-11T00:35:55.279481119Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DmS3IetBwqoZFY34Xk3Ohek","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.284473_564.1","section":"bucket.instance","name":"ufpbfn-38:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.38","timestamp":"2021-02-11T00:35:55.284473066Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DmS3IetBwqoZFY34Xk3Ohek","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003755.286540_565.1","section":"bucket","name":"ufpbfn-38","timestamp":"2021-02-11T00:35:55.286540135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0ru7BE0bzAXx0-Pb22dsKmU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003755.291866_566.1","section":"bucket","name":"ufpbfn-38","timestamp":"2021-02-11T00:35:55.291866653Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0ru7BE0bzAXx0-Pb22dsKmU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.788416_570.1","section":"bucket.instance","name":"ufpbfn-39:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.39","timestamp":"2021-02-11T00:35:57.788416591Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1vgWQA2CxVJU7vO6qXuqW2w","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.796322_571.1","section":"bucket.instance","name":"ufpbfn-39:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.39","timestamp":"2021-02-11T00:35:57.796322629Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1vgWQA2CxVJU7vO6qXuqW2w","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.798608_572.1","section":"bucket","name":"ufpbfn-39","timestamp":"2021-02-11T00:35:57.798608517Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kSxSKKo8ruuBEqht7yFAiKh","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.804310_573.1","section":"bucket","name":"ufpbfn-39","timestamp":"2021-02-11T00:35:57.804310795Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kSxSKKo8ruuBEqht7yFAiKh","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.834923_574.1","section":"bucket.instance","name":"ufpbfn-40:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.40","timestamp":"2021-02-11T00:35:57.834923842Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jcrlr-QMPmHhG1khCqUK-HL","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.844273_575.1","section":"bucket.instance","name":"ufpbfn-40:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.40","timestamp":"2021-02-11T00:35:57.844273207Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jcrlr-QMPmHhG1khCqUK-HL","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.846725_576.1","section":"bucket","name":"ufpbfn-40","timestamp":"2021-02-11T00:35:57.846725006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-U2oP92YIGzVDnZTkauQDfU","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.851372_577.1","section":"bucket","name":"ufpbfn-40","timestamp":"2021-02-11T00:35:57.851372809Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-U2oP92YIGzVDnZTkauQDfU","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.893851_578.1","section":"bucket.instance","name":"ufpbfn-41:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.41","timestamp":"2021-02-11T00:35:57.893851494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_F6KHu-saDEKBY8ttlAXOWyy","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.902224_579.1","section":"bucket.instance","name":"ufpbfn-41:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.4340.41","timestamp":"2021-02-11T00:35:57.902224837Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_F6KHu-saDEKBY8ttlAXOWyy","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003757.904331_580.1","section":"bucket","name":"ufpbfn-41","timestamp":"2021-02-11T00:35:57.904331122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Hr9JbpKcJAgiTFrG2I388EO","ver":1},"status":{"status":"write"}}},{"id":"1_1613003757.913330_581.1","section":"bucket","name":"ufpbfn-41","timestamp":"2021-02-11T00:35:57.913330476Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Hr9JbpKcJAgiTFrG2I388EO","ver":1},"status":{"status":"complete"}}}] 2021-02-11T00:38:16.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '4d4a6517-ce7a-42dd-9c62-65c684a638fb', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:16.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '4d4a6517-ce7a-42dd-9c62-65c684a638fb', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:16.693 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 4d4a6517-ce7a-42dd-9c62-65c684a638fb --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:16.731 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:16.731 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:16.976 INFO:teuthology.orchestra.run.smithi057.stdout:[{"id":"1_1613003825.628764_191.1","section":"bucket.instance","name":"ufpbfn-42:fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605.5104.1","timestamp":"2021-02-11T00:37:05.628764942Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4tHwxy_FADQMuBVc12p1geJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613003825.635773_192.1","section":"bucket.instance","name":"ufpbfn-42:fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605.5104.1","timestamp":"2021-02-11T00:37:05.635773643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4tHwxy_FADQMuBVc12p1geJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003825.638122_193.1","section":"bucket","name":"ufpbfn-42","timestamp":"2021-02-11T00:37:05.638122300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P9BikqaytQGQAlleOwCn3X1","ver":1},"status":{"status":"write"}}},{"id":"1_1613003825.643903_194.1","section":"bucket","name":"ufpbfn-42","timestamp":"2021-02-11T00:37:05.643903159Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P9BikqaytQGQAlleOwCn3X1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003825.710077_195.1","section":"bucket.instance","name":"ufpbfn-43:fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605.5104.2","timestamp":"2021-02-11T00:37:05.710077452Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xfDOa4_Irp2TpLX_EJl2gid","ver":1},"status":{"status":"write"}}},{"id":"1_1613003825.718398_196.1","section":"bucket.instance","name":"ufpbfn-43:fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605.5104.2","timestamp":"2021-02-11T00:37:05.718398250Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xfDOa4_Irp2TpLX_EJl2gid","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003825.720326_197.1","section":"bucket","name":"ufpbfn-43","timestamp":"2021-02-11T00:37:05.720326392Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LYyAdLnuaQJc9RwoOyxOBAw","ver":1},"status":{"status":"write"}}},{"id":"1_1613003825.727943_198.1","section":"bucket","name":"ufpbfn-43","timestamp":"2021-02-11T00:37:05.727943466Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LYyAdLnuaQJc9RwoOyxOBAw","ver":1},"status":{"status":"complete"}}}] 2021-02-11T00:38:16.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '4d4a6517-ce7a-42dd-9c62-65c684a638fb', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:16.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '4d4a6517-ce7a-42dd-9c62-65c684a638fb', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:16.977 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 4d4a6517-ce7a-42dd-9c62-65c684a638fb --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:17.024 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:17.024 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:17.258 INFO:teuthology.orchestra.run.smithi095.stdout:[{"id":"1_1613003825.628764_191.1","section":"bucket.instance","name":"ufpbfn-42:fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605.5104.1","timestamp":"2021-02-11T00:37:05.628764942Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4tHwxy_FADQMuBVc12p1geJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613003825.635773_192.1","section":"bucket.instance","name":"ufpbfn-42:fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605.5104.1","timestamp":"2021-02-11T00:37:05.635773643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4tHwxy_FADQMuBVc12p1geJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003825.638122_193.1","section":"bucket","name":"ufpbfn-42","timestamp":"2021-02-11T00:37:05.638122300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P9BikqaytQGQAlleOwCn3X1","ver":1},"status":{"status":"write"}}},{"id":"1_1613003825.643903_194.1","section":"bucket","name":"ufpbfn-42","timestamp":"2021-02-11T00:37:05.643903159Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P9BikqaytQGQAlleOwCn3X1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003825.710077_195.1","section":"bucket.instance","name":"ufpbfn-43:fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605.5104.2","timestamp":"2021-02-11T00:37:05.710077452Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xfDOa4_Irp2TpLX_EJl2gid","ver":1},"status":{"status":"write"}}},{"id":"1_1613003825.718398_196.1","section":"bucket.instance","name":"ufpbfn-43:fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605.5104.2","timestamp":"2021-02-11T00:37:05.718398250Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xfDOa4_Irp2TpLX_EJl2gid","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003825.720326_197.1","section":"bucket","name":"ufpbfn-43","timestamp":"2021-02-11T00:37:05.720326392Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LYyAdLnuaQJc9RwoOyxOBAw","ver":1},"status":{"status":"write"}}},{"id":"1_1613003825.727943_198.1","section":"bucket","name":"ufpbfn-43","timestamp":"2021-02-11T00:37:05.727943466Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LYyAdLnuaQJc9RwoOyxOBAw","ver":1},"status":{"status":"complete"}}}] 2021-02-11T00:38:17.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '4d4a6517-ce7a-42dd-9c62-65c684a638fb', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:17.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '4d4a6517-ce7a-42dd-9c62-65c684a638fb', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:17.259 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 4d4a6517-ce7a-42dd-9c62-65c684a638fb --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:17.298 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:17.298 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:17.553 INFO:teuthology.orchestra.run.smithi057.stdout:[{"id":"1_1613003825.628764_191.1","section":"bucket.instance","name":"ufpbfn-42:fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605.5104.1","timestamp":"2021-02-11T00:37:05.628764942Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4tHwxy_FADQMuBVc12p1geJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613003825.635773_192.1","section":"bucket.instance","name":"ufpbfn-42:fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605.5104.1","timestamp":"2021-02-11T00:37:05.635773643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4tHwxy_FADQMuBVc12p1geJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003825.638122_193.1","section":"bucket","name":"ufpbfn-42","timestamp":"2021-02-11T00:37:05.638122300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P9BikqaytQGQAlleOwCn3X1","ver":1},"status":{"status":"write"}}},{"id":"1_1613003825.643903_194.1","section":"bucket","name":"ufpbfn-42","timestamp":"2021-02-11T00:37:05.643903159Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P9BikqaytQGQAlleOwCn3X1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003825.710077_195.1","section":"bucket.instance","name":"ufpbfn-43:fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605.5104.2","timestamp":"2021-02-11T00:37:05.710077452Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xfDOa4_Irp2TpLX_EJl2gid","ver":1},"status":{"status":"write"}}},{"id":"1_1613003825.718398_196.1","section":"bucket.instance","name":"ufpbfn-43:fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605.5104.2","timestamp":"2021-02-11T00:37:05.718398250Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xfDOa4_Irp2TpLX_EJl2gid","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003825.720326_197.1","section":"bucket","name":"ufpbfn-43","timestamp":"2021-02-11T00:37:05.720326392Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LYyAdLnuaQJc9RwoOyxOBAw","ver":1},"status":{"status":"write"}}},{"id":"1_1613003825.727943_198.1","section":"bucket","name":"ufpbfn-43","timestamp":"2021-02-11T00:37:05.727943466Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LYyAdLnuaQJc9RwoOyxOBAw","ver":1},"status":{"status":"complete"}}}] 2021-02-11T00:38:17.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '4d4a6517-ce7a-42dd-9c62-65c684a638fb', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:17.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '4d4a6517-ce7a-42dd-9c62-65c684a638fb', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:17.554 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 4d4a6517-ce7a-42dd-9c62-65c684a638fb --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:17.593 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:17.594 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:17.837 INFO:teuthology.orchestra.run.smithi095.stdout:[{"id":"1_1613003825.628764_191.1","section":"bucket.instance","name":"ufpbfn-42:fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605.5104.1","timestamp":"2021-02-11T00:37:05.628764942Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4tHwxy_FADQMuBVc12p1geJ","ver":1},"status":{"status":"write"}}},{"id":"1_1613003825.635773_192.1","section":"bucket.instance","name":"ufpbfn-42:fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605.5104.1","timestamp":"2021-02-11T00:37:05.635773643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4tHwxy_FADQMuBVc12p1geJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003825.638122_193.1","section":"bucket","name":"ufpbfn-42","timestamp":"2021-02-11T00:37:05.638122300Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P9BikqaytQGQAlleOwCn3X1","ver":1},"status":{"status":"write"}}},{"id":"1_1613003825.643903_194.1","section":"bucket","name":"ufpbfn-42","timestamp":"2021-02-11T00:37:05.643903159Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P9BikqaytQGQAlleOwCn3X1","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003825.710077_195.1","section":"bucket.instance","name":"ufpbfn-43:fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605.5104.2","timestamp":"2021-02-11T00:37:05.710077452Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xfDOa4_Irp2TpLX_EJl2gid","ver":1},"status":{"status":"write"}}},{"id":"1_1613003825.718398_196.1","section":"bucket.instance","name":"ufpbfn-43:fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605.5104.2","timestamp":"2021-02-11T00:37:05.718398250Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xfDOa4_Irp2TpLX_EJl2gid","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003825.720326_197.1","section":"bucket","name":"ufpbfn-43","timestamp":"2021-02-11T00:37:05.720326392Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LYyAdLnuaQJc9RwoOyxOBAw","ver":1},"status":{"status":"write"}}},{"id":"1_1613003825.727943_198.1","section":"bucket","name":"ufpbfn-43","timestamp":"2021-02-11T00:37:05.727943466Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LYyAdLnuaQJc9RwoOyxOBAw","ver":1},"status":{"status":"complete"}}}] 2021-02-11T00:38:17.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '702c963a-2caf-4007-8190-7df3d99bc8ac', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:17.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '702c963a-2caf-4007-8190-7df3d99bc8ac', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:17.838 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 702c963a-2caf-4007-8190-7df3d99bc8ac --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:17.879 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:17.880 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:18.126 INFO:teuthology.orchestra.run.smithi057.stdout:[{"id":"1_1613003886.872200_192.1","section":"bucket.instance","name":"ufpbfn-44:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.5146.1","timestamp":"2021-02-11T00:38:06.872200032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gRhssqWt1hPqX1KdR_-pl5M","ver":1},"status":{"status":"write"}}},{"id":"1_1613003886.878212_193.1","section":"bucket.instance","name":"ufpbfn-44:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.5146.1","timestamp":"2021-02-11T00:38:06.878212315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gRhssqWt1hPqX1KdR_-pl5M","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003886.880511_194.1","section":"bucket","name":"ufpbfn-44","timestamp":"2021-02-11T00:38:06.880511358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YF98CHcCtHPB-S3qYl5Wlpx","ver":1},"status":{"status":"write"}}},{"id":"1_1613003886.885776_195.1","section":"bucket","name":"ufpbfn-44","timestamp":"2021-02-11T00:38:06.885776049Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YF98CHcCtHPB-S3qYl5Wlpx","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003886.925954_196.1","section":"bucket.instance","name":"ufpbfn-45:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.5146.2","timestamp":"2021-02-11T00:38:06.925954972Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tziqRo2FNovoIU3WQBn9lJb","ver":1},"status":{"status":"write"}}},{"id":"1_1613003886.931537_197.1","section":"bucket.instance","name":"ufpbfn-45:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.5146.2","timestamp":"2021-02-11T00:38:06.931537010Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tziqRo2FNovoIU3WQBn9lJb","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003886.933743_198.1","section":"bucket","name":"ufpbfn-45","timestamp":"2021-02-11T00:38:06.933743038Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pv-kSV6s1iTNkdo8OU2zbJG","ver":1},"status":{"status":"write"}}},{"id":"1_1613003886.939234_199.1","section":"bucket","name":"ufpbfn-45","timestamp":"2021-02-11T00:38:06.939234931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pv-kSV6s1iTNkdo8OU2zbJG","ver":1},"status":{"status":"complete"}}}] 2021-02-11T00:38:18.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '702c963a-2caf-4007-8190-7df3d99bc8ac', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:18.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '702c963a-2caf-4007-8190-7df3d99bc8ac', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:18.127 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 702c963a-2caf-4007-8190-7df3d99bc8ac --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:18.164 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:18.164 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:18.414 INFO:teuthology.orchestra.run.smithi095.stdout:[{"id":"1_1613003886.872200_192.1","section":"bucket.instance","name":"ufpbfn-44:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.5146.1","timestamp":"2021-02-11T00:38:06.872200032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gRhssqWt1hPqX1KdR_-pl5M","ver":1},"status":{"status":"write"}}},{"id":"1_1613003886.878212_193.1","section":"bucket.instance","name":"ufpbfn-44:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.5146.1","timestamp":"2021-02-11T00:38:06.878212315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gRhssqWt1hPqX1KdR_-pl5M","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003886.880511_194.1","section":"bucket","name":"ufpbfn-44","timestamp":"2021-02-11T00:38:06.880511358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YF98CHcCtHPB-S3qYl5Wlpx","ver":1},"status":{"status":"write"}}},{"id":"1_1613003886.885776_195.1","section":"bucket","name":"ufpbfn-44","timestamp":"2021-02-11T00:38:06.885776049Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YF98CHcCtHPB-S3qYl5Wlpx","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003886.925954_196.1","section":"bucket.instance","name":"ufpbfn-45:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.5146.2","timestamp":"2021-02-11T00:38:06.925954972Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tziqRo2FNovoIU3WQBn9lJb","ver":1},"status":{"status":"write"}}},{"id":"1_1613003886.931537_197.1","section":"bucket.instance","name":"ufpbfn-45:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.5146.2","timestamp":"2021-02-11T00:38:06.931537010Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tziqRo2FNovoIU3WQBn9lJb","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003886.933743_198.1","section":"bucket","name":"ufpbfn-45","timestamp":"2021-02-11T00:38:06.933743038Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pv-kSV6s1iTNkdo8OU2zbJG","ver":1},"status":{"status":"write"}}},{"id":"1_1613003886.939234_199.1","section":"bucket","name":"ufpbfn-45","timestamp":"2021-02-11T00:38:06.939234931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pv-kSV6s1iTNkdo8OU2zbJG","ver":1},"status":{"status":"complete"}}}] 2021-02-11T00:38:18.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '702c963a-2caf-4007-8190-7df3d99bc8ac', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:18.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '702c963a-2caf-4007-8190-7df3d99bc8ac', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:18.415 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 702c963a-2caf-4007-8190-7df3d99bc8ac --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:18.455 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:18.455 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:18.717 INFO:teuthology.orchestra.run.smithi057.stdout:[{"id":"1_1613003886.872200_192.1","section":"bucket.instance","name":"ufpbfn-44:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.5146.1","timestamp":"2021-02-11T00:38:06.872200032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gRhssqWt1hPqX1KdR_-pl5M","ver":1},"status":{"status":"write"}}},{"id":"1_1613003886.878212_193.1","section":"bucket.instance","name":"ufpbfn-44:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.5146.1","timestamp":"2021-02-11T00:38:06.878212315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gRhssqWt1hPqX1KdR_-pl5M","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003886.880511_194.1","section":"bucket","name":"ufpbfn-44","timestamp":"2021-02-11T00:38:06.880511358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YF98CHcCtHPB-S3qYl5Wlpx","ver":1},"status":{"status":"write"}}},{"id":"1_1613003886.885776_195.1","section":"bucket","name":"ufpbfn-44","timestamp":"2021-02-11T00:38:06.885776049Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YF98CHcCtHPB-S3qYl5Wlpx","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003886.925954_196.1","section":"bucket.instance","name":"ufpbfn-45:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.5146.2","timestamp":"2021-02-11T00:38:06.925954972Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tziqRo2FNovoIU3WQBn9lJb","ver":1},"status":{"status":"write"}}},{"id":"1_1613003886.931537_197.1","section":"bucket.instance","name":"ufpbfn-45:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.5146.2","timestamp":"2021-02-11T00:38:06.931537010Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tziqRo2FNovoIU3WQBn9lJb","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003886.933743_198.1","section":"bucket","name":"ufpbfn-45","timestamp":"2021-02-11T00:38:06.933743038Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pv-kSV6s1iTNkdo8OU2zbJG","ver":1},"status":{"status":"write"}}},{"id":"1_1613003886.939234_199.1","section":"bucket","name":"ufpbfn-45","timestamp":"2021-02-11T00:38:06.939234931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pv-kSV6s1iTNkdo8OU2zbJG","ver":1},"status":{"status":"complete"}}}] 2021-02-11T00:38:18.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '702c963a-2caf-4007-8190-7df3d99bc8ac', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:18.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '702c963a-2caf-4007-8190-7df3d99bc8ac', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:18.717 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 702c963a-2caf-4007-8190-7df3d99bc8ac --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:18.756 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:18.756 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:18.997 INFO:teuthology.orchestra.run.smithi095.stdout:[{"id":"1_1613003886.872200_192.1","section":"bucket.instance","name":"ufpbfn-44:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.5146.1","timestamp":"2021-02-11T00:38:06.872200032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gRhssqWt1hPqX1KdR_-pl5M","ver":1},"status":{"status":"write"}}},{"id":"1_1613003886.878212_193.1","section":"bucket.instance","name":"ufpbfn-44:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.5146.1","timestamp":"2021-02-11T00:38:06.878212315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gRhssqWt1hPqX1KdR_-pl5M","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003886.880511_194.1","section":"bucket","name":"ufpbfn-44","timestamp":"2021-02-11T00:38:06.880511358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YF98CHcCtHPB-S3qYl5Wlpx","ver":1},"status":{"status":"write"}}},{"id":"1_1613003886.885776_195.1","section":"bucket","name":"ufpbfn-44","timestamp":"2021-02-11T00:38:06.885776049Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YF98CHcCtHPB-S3qYl5Wlpx","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003886.925954_196.1","section":"bucket.instance","name":"ufpbfn-45:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.5146.2","timestamp":"2021-02-11T00:38:06.925954972Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tziqRo2FNovoIU3WQBn9lJb","ver":1},"status":{"status":"write"}}},{"id":"1_1613003886.931537_197.1","section":"bucket.instance","name":"ufpbfn-45:e989c9f2-9c89-43f5-a54f-e1d0705a9abe.5146.2","timestamp":"2021-02-11T00:38:06.931537010Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tziqRo2FNovoIU3WQBn9lJb","ver":1},"status":{"status":"complete"}}},{"id":"1_1613003886.933743_198.1","section":"bucket","name":"ufpbfn-45","timestamp":"2021-02-11T00:38:06.933743038Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pv-kSV6s1iTNkdo8OU2zbJG","ver":1},"status":{"status":"write"}}},{"id":"1_1613003886.939234_199.1","section":"bucket","name":"ufpbfn-45","timestamp":"2021-02-11T00:38:06.939234931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pv-kSV6s1iTNkdo8OU2zbJG","ver":1},"status":{"status":"complete"}}}] 2021-02-11T00:38:18.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:18.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:18.998 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:38:19.036 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:19.036 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:19.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:19.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:19.344 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:38:19.381 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:19.381 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:19.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:19.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:19.799 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:38:19.837 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:19.837 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:20.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:20.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:20.105 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:38:20.138 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:20.138 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:20.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '7f3547bf-73e3-43a7-aa91-648db0f982f6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:20.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '7f3547bf-73e3-43a7-aa91-648db0f982f6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:20.504 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 7f3547bf-73e3-43a7-aa91-648db0f982f6 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:20.544 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:20.544 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:20.795 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2021-02-11T00:38:20.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '7f3547bf-73e3-43a7-aa91-648db0f982f6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:20.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '7f3547bf-73e3-43a7-aa91-648db0f982f6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:20.796 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 7f3547bf-73e3-43a7-aa91-648db0f982f6 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:20.837 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:20.837 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:21.096 INFO:teuthology.orchestra.run.smithi095.stdout:[] 2021-02-11T00:38:21.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '7f3547bf-73e3-43a7-aa91-648db0f982f6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:21.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '7f3547bf-73e3-43a7-aa91-648db0f982f6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:21.096 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 7f3547bf-73e3-43a7-aa91-648db0f982f6 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:21.143 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:21.144 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:21.416 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2021-02-11T00:38:21.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '7f3547bf-73e3-43a7-aa91-648db0f982f6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:21.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '7f3547bf-73e3-43a7-aa91-648db0f982f6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:21.417 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 7f3547bf-73e3-43a7-aa91-648db0f982f6 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:21.459 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:21.459 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:21.715 INFO:teuthology.orchestra.run.smithi095.stdout:[] 2021-02-11T00:38:21.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '4d4a6517-ce7a-42dd-9c62-65c684a638fb', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:21.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '4d4a6517-ce7a-42dd-9c62-65c684a638fb', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:21.716 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 4d4a6517-ce7a-42dd-9c62-65c684a638fb --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:21.756 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:21.999 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2021-02-11T00:38:21.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '4d4a6517-ce7a-42dd-9c62-65c684a638fb', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:21.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '4d4a6517-ce7a-42dd-9c62-65c684a638fb', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:22.000 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 4d4a6517-ce7a-42dd-9c62-65c684a638fb --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:22.039 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:22.039 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:22.290 INFO:teuthology.orchestra.run.smithi095.stdout:[] 2021-02-11T00:38:22.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '4d4a6517-ce7a-42dd-9c62-65c684a638fb', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:22.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '4d4a6517-ce7a-42dd-9c62-65c684a638fb', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:22.291 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 4d4a6517-ce7a-42dd-9c62-65c684a638fb --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:22.333 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:22.334 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:22.595 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2021-02-11T00:38:22.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '4d4a6517-ce7a-42dd-9c62-65c684a638fb', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:22.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '4d4a6517-ce7a-42dd-9c62-65c684a638fb', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:22.595 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 4d4a6517-ce7a-42dd-9c62-65c684a638fb --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:22.635 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:22.635 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:22.876 INFO:teuthology.orchestra.run.smithi095.stdout:[] 2021-02-11T00:38:22.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '702c963a-2caf-4007-8190-7df3d99bc8ac', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:22.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '702c963a-2caf-4007-8190-7df3d99bc8ac', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:22.877 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 702c963a-2caf-4007-8190-7df3d99bc8ac --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:22.912 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:22.912 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:23.155 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2021-02-11T00:38:23.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '702c963a-2caf-4007-8190-7df3d99bc8ac', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:23.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '702c963a-2caf-4007-8190-7df3d99bc8ac', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:23.156 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 702c963a-2caf-4007-8190-7df3d99bc8ac --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:23.195 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:23.195 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:23.448 INFO:teuthology.orchestra.run.smithi095.stdout:[] 2021-02-11T00:38:23.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '702c963a-2caf-4007-8190-7df3d99bc8ac', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:23.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '702c963a-2caf-4007-8190-7df3d99bc8ac', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:23.448 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 702c963a-2caf-4007-8190-7df3d99bc8ac --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:23.489 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:23.489 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:23.751 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2021-02-11T00:38:23.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '702c963a-2caf-4007-8190-7df3d99bc8ac', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:23.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '702c963a-2caf-4007-8190-7df3d99bc8ac', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:23.751 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 702c963a-2caf-4007-8190-7df3d99bc8ac --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:23.789 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:23.789 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:24.046 INFO:teuthology.orchestra.run.smithi095.stdout:[] 2021-02-11T00:38:24.047 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2021-02-11T00:38:24.048 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-46 2021-02-11T00:38:24.097 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-47 2021-02-11T00:38:24.173 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-48 2021-02-11T00:38:24.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:24.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:24.412 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:24.451 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:24.451 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:24.620 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T00:38:24.659 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613003904.221233_252.1","last_update":"2021-02-11T00:38:24.221233Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:38:24.659 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:38:24.659 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:24.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:24.660 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:24.699 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:24.699 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:24.945 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003904.221233_252.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:24.221233Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:38:24.945 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:38:24.945 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:38:24.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:24.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:24.946 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:24.988 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:24.988 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:25.249 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003904.221233_252.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:24.221233Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:38:25.250 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:38:25.250 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:38:25.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:25.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:25.251 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:25.290 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:25.290 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:25.520 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003904.221233_252.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:24.221233Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:38:25.520 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:38:25.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:25.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:25.521 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:25.563 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:25.563 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:25.840 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"00000000000000000000:00000000000000003171","last_update":"2021-02-11T00:38:24.503743Z"},{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T00:34:45.582829Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T00:35:57.944627Z"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T00:34:23.574930Z"}] 2021-02-11T00:38:25.840 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T00:38:25.840 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:25.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:25.841 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:25.885 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:25.885 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:26.128 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7281982967011226472},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003171","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:24.503743Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.582829Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000780","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:57.944627Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.574930Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T00:38:26.128 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-02-11T00:38:26.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:26.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:26.129 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:26.169 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:26.169 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:26.458 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"00000000000000000000:00000000000000003171","last_update":"2021-02-11T00:38:24.503743Z"},{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T00:34:45.582829Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T00:35:57.944627Z"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T00:34:23.574930Z"}] 2021-02-11T00:38:26.458 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T00:38:26.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:26.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:26.459 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:26.542 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:26.542 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:26.805 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13309572567137725785},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003171","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:24.503743Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.582829Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000780","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:57.944627Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.574930Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T00:38:26.806 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-02-11T00:38:26.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:26.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:26.806 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:26.899 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:26.899 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:27.142 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"00000000000000000000:00000000000000003171","last_update":"2021-02-11T00:38:24.503743Z"},{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T00:34:45.582829Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T00:35:57.944627Z"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T00:34:23.574930Z"}] 2021-02-11T00:38:27.142 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T00:38:27.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:27.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:27.143 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:27.185 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:27.185 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:27.420 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4767706083908344750},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003039","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:24.276506Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.582829Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000780","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:57.944627Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.574930Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T00:38:27.421 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003171 target=00000000000000000000:00000000000000003039 2021-02-11T00:38:32.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:32.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:32.422 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:32.464 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:32.464 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:32.708 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4767706083908344750},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003303","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:27.321596Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.582829Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000780","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:57.944627Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.574930Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T00:38:32.708 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-02-11T00:38:32.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:32.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:32.709 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:32.789 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:32.789 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:33.044 INFO:teuthology.orchestra.run.smithi095.stdout:[{"marker":"00000000000000000000:00000000000000003303","last_update":"2021-02-11T00:38:27.227762Z"},{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T00:34:45.572834Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T00:35:57.890782Z"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T00:34:23.563592Z"}] 2021-02-11T00:38:33.044 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T00:38:33.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:33.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:33.045 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:33.088 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:33.088 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:33.350 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":5992573881530547290},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003303","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:27.227762Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.572834Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000780","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:57.890782Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.563592Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T00:38:33.351 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-02-11T00:38:33.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:33.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:33.351 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:33.390 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:33.390 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:33.640 INFO:teuthology.orchestra.run.smithi095.stdout:[{"marker":"00000000000000000000:00000000000000003303","last_update":"2021-02-11T00:38:27.227762Z"},{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T00:34:45.572834Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T00:35:57.890782Z"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T00:34:23.563592Z"}] 2021-02-11T00:38:33.640 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T00:38:33.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:33.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:33.641 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:33.684 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:33.684 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:33.942 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12481821032201219352},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003303","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:27.227762Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.572834Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000780","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:57.890782Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.563592Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T00:38:33.942 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-02-11T00:38:33.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:33.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:33.943 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:34.023 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:34.024 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:34.304 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"00000000000000000000:00000000000000003303","last_update":"2021-02-11T00:38:24.557970Z"},{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T00:34:45.549594Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T00:35:57.736316Z"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T00:34:23.539931Z"}] 2021-02-11T00:38:34.304 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T00:38:34.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:34.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:34.305 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:34.392 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:34.392 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:34.643 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":15593642182000949264},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003303","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:24.557970Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.549594Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000780","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:57.736316Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.539931Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T00:38:34.643 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-02-11T00:38:34.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:34.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:34.644 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:34.724 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:34.724 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:35.003 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"00000000000000000000:00000000000000003303","last_update":"2021-02-11T00:38:24.557970Z"},{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T00:34:45.549594Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T00:35:57.736316Z"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T00:34:23.539931Z"}] 2021-02-11T00:38:35.003 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T00:38:35.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:35.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:35.004 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:35.051 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:35.052 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:35.277 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11510982620117777593},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000003303","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:24.557970Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000004635","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:45.549594Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000000780","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:35:57.736316Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"00000000000000000000:00000000000000002374","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:34:23.539931Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-02-11T00:38:35.278 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-02-11T00:38:35.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:35.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:35.278 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:35.315 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:35.315 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:35.595 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"00000000000000000000:00000000000000003303","last_update":"2021-02-11T00:38:27.321596Z"},{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T00:34:45.582829Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T00:35:57.944627Z"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T00:34:23.574930Z"}] 2021-02-11T00:38:35.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:35.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:35.596 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:38:35.673 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:35.673 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:35.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:35.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:35.950 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:35.987 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:35.987 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:36.251 INFO:teuthology.orchestra.run.smithi095.stdout:[{"marker":"00000000000000000000:00000000000000003303","last_update":"2021-02-11T00:38:27.227762Z"},{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T00:34:45.572834Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T00:35:57.890782Z"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T00:34:23.563592Z"}] 2021-02-11T00:38:36.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:36.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:36.251 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:38:36.327 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:36.327 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:36.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:36.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:38:36.614 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:38:36.654 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:36.654 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:36.928 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"00000000000000000000:00000000000000003303","last_update":"2021-02-11T00:38:27.321596Z"},{"marker":"00000000000000000000:00000000000000004635","last_update":"2021-02-11T00:34:45.582829Z"},{"marker":"00000000000000000000:00000000000000000780","last_update":"2021-02-11T00:35:57.944627Z"},{"marker":"00000000000000000000:00000000000000002374","last_update":"2021-02-11T00:34:23.574930Z"}] 2021-02-11T00:38:36.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:36.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:36.928 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:38:37.000 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:37.000 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:37.279 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2021-02-11T00:38:37.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:37.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:37.280 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:38:37.321 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:37.321 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:37.352 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:38:37.350+0000 7f08d9f4f080 0 failed reading obj info from .rgw.root:zone_info.35e954f9-5ea0-411e-8e41-e821d2787b31: (2) No such file or directory 2021-02-11T00:38:37.352 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:38:37.350+0000 7f08d9f4f080 0 WARNING: could not read zone params for zone id=35e954f9-5ea0-411e-8e41-e821d2787b31 name=test-zone3 2021-02-11T00:38:37.353 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:38:37.350+0000 7f08d9f4f080 0 failed reading obj info from .rgw.root:zone_info.e989c9f2-9c89-43f5-a54f-e1d0705a9abe: (2) No such file or directory 2021-02-11T00:38:37.353 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:38:37.350+0000 7f08d9f4f080 0 WARNING: could not read zone params for zone id=e989c9f2-9c89-43f5-a54f-e1d0705a9abe name=test-zone1 2021-02-11T00:38:37.372 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"CP2FO0ORQU6DSQKV","secret_key":"szUDueomrwAKvai4rUGz9pCzuDdncCWD"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T00:38:37.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:37.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:38:37.373 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:38:37.453 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:38:37.453 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:38:37.714 INFO:teuthology.orchestra.run.smithi095.stderr:Sending period to new master zone e989c9f2-9c89-43f5-a54f-e1d0705a9abe 2021-02-11T00:38:37.814 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","epoch":2,"predecessor_uuid":"4d4a6517-ce7a-42dd-9c62-65c684a638fb","sync_status":["","1_1613003825.727943_198.1","",""],"period_map":{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","zones":[{"id":"35e954f9-5ea0-411e-8e41-e821d2787b31","name":"test-zone3","endpoints":["http://smithi057.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","name":"test-zone4","endpoints":["http://smithi095.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","endpoints":["http://smithi095.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"35e954f9-5ea0-411e-8e41-e821d2787b31","val":398828966},{"key":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","val":1248431312},{"key":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","val":3133685257},{"key":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","val":1241439546}]},"master_zonegroup":"test-zonegroup","master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","realm_name":"test-realm","realm_epoch":4} 2021-02-11T00:38:37.814 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2021-02-11T00:39:37.815 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-49 2021-02-11T00:39:37.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:39:37.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:39:37.920 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:39:37.959 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:39:37.959 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:39:38.165 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T00:39:38.203 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613003977.863186_285.1","last_update":"2021-02-11T00:39:37.863186Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:39:38.203 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:39:38.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:39:38.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:39:38.204 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:39:38.245 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:39:38.245 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:39:38.498 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003904.221233_252.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:24.221233Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:39:38.498 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613003977.863186_285.1 target=1_1613003904.221233_252.1 2021-02-11T00:39:43.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:39:43.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:39:43.499 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:39:43.540 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:39:43.541 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:39:43.803 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003904.221233_252.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:24.221233Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:39:43.803 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613003977.863186_285.1 target=1_1613003904.221233_252.1 2021-02-11T00:39:48.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:39:48.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:39:48.805 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:39:48.845 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:39:48.845 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:39:49.095 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003904.221233_252.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:24.221233Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:39:49.095 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613003977.863186_285.1 target=1_1613003904.221233_252.1 2021-02-11T00:39:54.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:39:54.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:39:54.096 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:39:54.135 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:39:54.135 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:39:54.393 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003904.221233_252.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:24.221233Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:39:54.393 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613003977.863186_285.1 target=1_1613003904.221233_252.1 2021-02-11T00:39:59.394 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:39:59.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:39:59.395 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:39:59.434 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:39:59.434 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:39:59.695 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003904.221233_252.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:24.221233Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:39:59.695 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613003977.863186_285.1 target=1_1613003904.221233_252.1 2021-02-11T00:40:04.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:40:04.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:40:04.696 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:40:04.737 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:40:04.737 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:40:05.000 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003904.221233_252.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:38:24.221233Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:40:05.001 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613003977.863186_285.1 target=1_1613003904.221233_252.1 2021-02-11T00:40:08.063 INFO:tasks.rgw.c1.client.1.smithi057.stdout:2021-02-11T00:40:08.059+0000 7f6363fff700 -1 data sync zone:e989c9f2 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-02-11T00:40:10.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:40:10.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:40:10.002 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:40:10.042 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:40:10.043 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:40:10.291 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:40:10.292 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:40:10.292 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:40:10.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:40:10.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:40:10.292 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:40:10.332 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:40:10.332 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:40:10.599 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:40:10.600 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:40:10.600 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:40:10.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:40:10.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:40:10.600 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:40:10.642 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:40:10.642 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:40:10.875 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:40:10.875 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:40:10.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=e989c9f2-9c89-43f5-a54f-e1d0705a9abe', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:40:10.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=e989c9f2-9c89-43f5-a54f-e1d0705a9abe', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:40:10.892 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=e989c9f2-9c89-43f5-a54f-e1d0705a9abe --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:40:10.953 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:40:10.953 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:40:10.983 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:40:10.977+0000 7f8769679080 0 failed reading obj info from .rgw.root:zone_info.35e954f9-5ea0-411e-8e41-e821d2787b31: (2) No such file or directory 2021-02-11T00:40:10.983 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:40:10.977+0000 7f8769679080 0 WARNING: could not read zone params for zone id=35e954f9-5ea0-411e-8e41-e821d2787b31 name=test-zone3 2021-02-11T00:40:10.985 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:40:10.981+0000 7f8769679080 0 failed reading obj info from .rgw.root:zone_info.e989c9f2-9c89-43f5-a54f-e1d0705a9abe: (2) No such file or directory 2021-02-11T00:40:10.985 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:40:10.981+0000 7f8769679080 0 WARNING: could not read zone params for zone id=e989c9f2-9c89-43f5-a54f-e1d0705a9abe name=test-zone1 2021-02-11T00:40:11.000 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"CP2FO0ORQU6DSQKV","secret_key":"szUDueomrwAKvai4rUGz9pCzuDdncCWD"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T00:40:11.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:40:11.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:40:11.001 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:40:11.081 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:40:11.082 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:40:11.328 INFO:teuthology.orchestra.run.smithi095.stderr:Sending period to new master zone e989c9f2-9c89-43f5-a54f-e1d0705a9abe 2021-02-11T00:40:11.437 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","epoch":3,"predecessor_uuid":"4d4a6517-ce7a-42dd-9c62-65c684a638fb","sync_status":["","1_1613003825.727943_198.1","",""],"period_map":{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","zones":[{"id":"35e954f9-5ea0-411e-8e41-e821d2787b31","name":"test-zone3","endpoints":["http://smithi057.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","name":"test-zone4","endpoints":["http://smithi095.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","endpoints":["http://smithi095.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe"}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"35e954f9-5ea0-411e-8e41-e821d2787b31","val":398828966},{"key":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","val":1248431312},{"key":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","val":3133685257},{"key":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","val":1241439546}]},"master_zonegroup":"test-zonegroup","master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","realm_name":"test-realm","realm_epoch":4} 2021-02-11T00:40:11.438 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "e989c9f2-9c89-43f5-a54f-e1d0705a9abe" 2021-02-11T00:41:11.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:41:11.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:41:11.799 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:41:11.841 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:41:11.841 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:41:11.867 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:41:11.860+0000 7f6eae306080 0 failed reading obj info from .rgw.root:zone_info.35e954f9-5ea0-411e-8e41-e821d2787b31: (2) No such file or directory 2021-02-11T00:41:11.867 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:41:11.860+0000 7f6eae306080 0 WARNING: could not read zone params for zone id=35e954f9-5ea0-411e-8e41-e821d2787b31 name=test-zone3 2021-02-11T00:41:11.868 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:41:11.864+0000 7f6eae306080 0 failed reading obj info from .rgw.root:zone_info.e989c9f2-9c89-43f5-a54f-e1d0705a9abe: (2) No such file or directory 2021-02-11T00:41:11.868 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:41:11.864+0000 7f6eae306080 0 WARNING: could not read zone params for zone id=e989c9f2-9c89-43f5-a54f-e1d0705a9abe name=test-zone1 2021-02-11T00:41:11.881 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"CP2FO0ORQU6DSQKV","secret_key":"szUDueomrwAKvai4rUGz9pCzuDdncCWD"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T00:41:11.882 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:41:11.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:41:11.882 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:41:11.967 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:41:11.968 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:41:12.195 INFO:teuthology.orchestra.run.smithi095.stderr:Sending period to new master zone e989c9f2-9c89-43f5-a54f-e1d0705a9abe 2021-02-11T00:41:12.294 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","epoch":4,"predecessor_uuid":"4d4a6517-ce7a-42dd-9c62-65c684a638fb","sync_status":["","1_1613003825.727943_198.1","",""],"period_map":{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","zones":[{"id":"35e954f9-5ea0-411e-8e41-e821d2787b31","name":"test-zone3","endpoints":["http://smithi057.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","name":"test-zone4","endpoints":["http://smithi095.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","endpoints":["http://smithi095.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe"}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"35e954f9-5ea0-411e-8e41-e821d2787b31","val":398828966},{"key":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","val":1248431312},{"key":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","val":3133685257},{"key":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","val":1241439546}]},"master_zonegroup":"test-zonegroup","master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","realm_name":"test-realm","realm_epoch":4} 2021-02-11T00:41:12.295 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2021-02-11T00:42:12.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:42:12.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:42:12.297 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:42:12.338 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:42:12.338 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:42:12.367 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:42:12.363+0000 7f5858625080 0 failed reading obj info from .rgw.root:zone_info.35e954f9-5ea0-411e-8e41-e821d2787b31: (2) No such file or directory 2021-02-11T00:42:12.367 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:42:12.363+0000 7f5858625080 0 WARNING: could not read zone params for zone id=35e954f9-5ea0-411e-8e41-e821d2787b31 name=test-zone3 2021-02-11T00:42:12.368 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:42:12.367+0000 7f5858625080 0 failed reading obj info from .rgw.root:zone_info.e989c9f2-9c89-43f5-a54f-e1d0705a9abe: (2) No such file or directory 2021-02-11T00:42:12.369 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:42:12.367+0000 7f5858625080 0 WARNING: could not read zone params for zone id=e989c9f2-9c89-43f5-a54f-e1d0705a9abe name=test-zone1 2021-02-11T00:42:12.383 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"CP2FO0ORQU6DSQKV","secret_key":"szUDueomrwAKvai4rUGz9pCzuDdncCWD"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","notif_pool":"test-zone2.rgw.log:notif"} 2021-02-11T00:42:12.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:42:12.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:42:12.384 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:42:12.467 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:42:12.468 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:42:12.716 INFO:teuthology.orchestra.run.smithi095.stderr:Sending period to new master zone e989c9f2-9c89-43f5-a54f-e1d0705a9abe 2021-02-11T00:42:12.832 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","epoch":5,"predecessor_uuid":"4d4a6517-ce7a-42dd-9c62-65c684a638fb","sync_status":["","1_1613003825.727943_198.1","",""],"period_map":{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","zones":[{"id":"35e954f9-5ea0-411e-8e41-e821d2787b31","name":"test-zone3","endpoints":["http://smithi057.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","name":"test-zone4","endpoints":["http://smithi095.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","endpoints":["http://smithi095.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"35e954f9-5ea0-411e-8e41-e821d2787b31","val":398828966},{"key":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","val":1248431312},{"key":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","val":3133685257},{"key":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","val":1241439546}]},"master_zonegroup":"test-zonegroup","master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","realm_name":"test-realm","realm_epoch":4} 2021-02-11T00:42:12.833 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2021-02-11T00:43:12.835 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2021-02-11T00:43:12.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:12.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:12.837 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:43:12.880 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:12.880 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:13.104 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T00:43:13.146 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613003977.863186_285.1","last_update":"2021-02-11T00:39:37.863186Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:43:13.146 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:43:13.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:13.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:13.147 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:43:13.185 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:13.185 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:13.434 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:43:13.435 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:43:13.435 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:43:13.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:13.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:13.435 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:43:13.476 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:13.477 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:13.739 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:43:13.739 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:43:13.739 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:43:13.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:13.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:13.740 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:43:13.781 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:13.781 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:14.019 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:43:14.020 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-02-11T00:43:14.020 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:14.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'CP2FO0ORQU6DSQKV', '--secret', 'szUDueomrwAKvai4rUGz9pCzuDdncCWD', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:14.020 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key CP2FO0ORQU6DSQKV --secret szUDueomrwAKvai4rUGz9pCzuDdncCWD --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:43:14.104 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:14.104 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:14.173 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:43:14.170+0000 7f2899e86080 0 failed reading obj info from .rgw.root:zone_info.35e954f9-5ea0-411e-8e41-e821d2787b31: (2) No such file or directory 2021-02-11T00:43:14.173 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:43:14.170+0000 7f2899e86080 0 WARNING: could not read zone params for zone id=35e954f9-5ea0-411e-8e41-e821d2787b31 name=test-zone3 2021-02-11T00:43:14.175 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:43:14.170+0000 7f2899e86080 0 failed reading obj info from .rgw.root:zone_info.e989c9f2-9c89-43f5-a54f-e1d0705a9abe: (2) No such file or directory 2021-02-11T00:43:14.175 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:43:14.170+0000 7f2899e86080 0 WARNING: could not read zone params for zone id=e989c9f2-9c89-43f5-a54f-e1d0705a9abe name=test-zone1 2021-02-11T00:43:14.184 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"87a71c00-15fb-44d3-9eec-a9e753aa755e","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"CP2FO0ORQU6DSQKV","secret_key":"szUDueomrwAKvai4rUGz9pCzuDdncCWD"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","notif_pool":"remove.rgw.log:notif"} 2021-02-11T00:43:14.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:14.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:14.185 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:43:14.266 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:14.267 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:22.408 INFO:teuthology.orchestra.run.smithi095.stderr:Sending period to new master zone e989c9f2-9c89-43f5-a54f-e1d0705a9abe 2021-02-11T00:43:22.507 INFO:teuthology.orchestra.run.smithi095.stdout:{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","epoch":6,"predecessor_uuid":"4d4a6517-ce7a-42dd-9c62-65c684a638fb","sync_status":["","1_1613003825.727943_198.1","",""],"period_map":{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","zones":[{"id":"35e954f9-5ea0-411e-8e41-e821d2787b31","name":"test-zone3","endpoints":["http://smithi057.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"87a71c00-15fb-44d3-9eec-a9e753aa755e","name":"remove","endpoints":[],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","name":"test-zone4","endpoints":["http://smithi095.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","endpoints":["http://smithi095.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"35e954f9-5ea0-411e-8e41-e821d2787b31","val":398828966},{"key":"87a71c00-15fb-44d3-9eec-a9e753aa755e","val":3594968091},{"key":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","val":1248431312},{"key":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","val":3133685257},{"key":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","val":1241439546}]},"master_zonegroup":"test-zonegroup","master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","realm_name":"test-realm","realm_epoch":4} 2021-02-11T00:43:22.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:22.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:22.508 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:43:22.553 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:22.553 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:22.576 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:43:22.576+0000 7fa45ecc3080 0 failed reading obj info from .rgw.root:zone_info.d88fea79-91d3-40e7-8bb7-515bb1e5c3b3: (2) No such file or directory 2021-02-11T00:43:22.576 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:43:22.576+0000 7fa45ecc3080 0 WARNING: could not read zone params for zone id=d88fea79-91d3-40e7-8bb7-515bb1e5c3b3 name=test-zone4 2021-02-11T00:43:22.578 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:43:22.576+0000 7fa45ecc3080 0 failed reading obj info from .rgw.root:zone_info.fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605: (2) No such file or directory 2021-02-11T00:43:22.578 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:43:22.576+0000 7fa45ecc3080 0 WARNING: could not read zone params for zone id=fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605 name=test-zone2 2021-02-11T00:43:22.587 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","zones":[{"id":"35e954f9-5ea0-411e-8e41-e821d2787b31","name":"test-zone3","endpoints":["http://smithi057.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","name":"test-zone4","endpoints":["http://smithi095.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","endpoints":["http://smithi095.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}} 2021-02-11T00:43:22.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:22.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:22.588 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:43:22.630 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:22.630 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:22.647 INFO:teuthology.orchestra.run.smithi057.stderr:zone name remove not found in zonegroup test-zonegroup 2021-02-11T00:43:22.655 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-11T00:43:22.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:22.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:22.655 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2021-02-11T00:43:22.692 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:22.692 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:22.732 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:43:22.730+0000 7fa0fc138080 0 failed reading obj info from .rgw.root:zone_info.35e954f9-5ea0-411e-8e41-e821d2787b31: (2) No such file or directory 2021-02-11T00:43:22.732 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:43:22.730+0000 7fa0fc138080 0 WARNING: could not read zone params for zone id=35e954f9-5ea0-411e-8e41-e821d2787b31 name=test-zone3 2021-02-11T00:43:22.733 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:43:22.730+0000 7fa0fc138080 0 failed reading obj info from .rgw.root:zone_info.e989c9f2-9c89-43f5-a54f-e1d0705a9abe: (2) No such file or directory 2021-02-11T00:43:22.733 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:43:22.730+0000 7fa0fc138080 0 WARNING: could not read zone params for zone id=e989c9f2-9c89-43f5-a54f-e1d0705a9abe name=test-zone1 2021-02-11T00:43:22.738 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T00:43:22.734+0000 7fa0fc138080 0 zone id 87a71c00-15fb-44d3-9eec-a9e753aa755e is not a part of zonegroup default 2021-02-11T00:43:22.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:22.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:22.749 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:43:22.805 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:22.805 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:22.874 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:43:22.872+0000 7fa985630080 0 WARNING: can't generate connection for zone 87a71c00-15fb-44d3-9eec-a9e753aa755e id remove: no endpoints defined 2021-02-11T00:43:23.203 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","epoch":7,"predecessor_uuid":"4d4a6517-ce7a-42dd-9c62-65c684a638fb","sync_status":["","1_1613003825.727943_198.1","",""],"period_map":{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","zones":[{"id":"35e954f9-5ea0-411e-8e41-e821d2787b31","name":"test-zone3","endpoints":["http://smithi057.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","name":"test-zone4","endpoints":["http://smithi095.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","endpoints":["http://smithi095.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"35e954f9-5ea0-411e-8e41-e821d2787b31","val":398828966},{"key":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","val":1248431312},{"key":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","val":3133685257},{"key":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","val":1241439546}]},"master_zonegroup":"test-zonegroup","master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","realm_name":"test-realm","realm_epoch":4} 2021-02-11T00:43:23.203 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2021-02-11T00:43:23.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:23.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:23.204 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:43:23.244 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:23.244 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:23.277 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"14938a1d-942e-4414-b54b-6e741f7f7888","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}} 2021-02-11T00:43:23.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:23.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:23.278 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:43:23.362 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:23.362 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:23.423 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"7ef30163-aa7c-4a91-ad66-1f645b4c0479","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","notif_pool":"remove.rgw.log:notif"} 2021-02-11T00:43:23.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:23.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:23.424 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:43:23.505 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:23.506 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:23.781 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","epoch":8,"predecessor_uuid":"4d4a6517-ce7a-42dd-9c62-65c684a638fb","sync_status":["","1_1613003825.727943_198.1","",""],"period_map":{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","zonegroups":[{"id":"14938a1d-942e-4414-b54b-6e741f7f7888","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"7ef30163-aa7c-4a91-ad66-1f645b4c0479","zones":[{"id":"7ef30163-aa7c-4a91-ad66-1f645b4c0479","name":"remove","endpoints":[],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","zones":[{"id":"35e954f9-5ea0-411e-8e41-e821d2787b31","name":"test-zone3","endpoints":["http://smithi057.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","name":"test-zone4","endpoints":["http://smithi095.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","endpoints":["http://smithi095.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"35e954f9-5ea0-411e-8e41-e821d2787b31","val":398828966},{"key":"7ef30163-aa7c-4a91-ad66-1f645b4c0479","val":887492302},{"key":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","val":1248431312},{"key":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","val":3133685257},{"key":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","val":1241439546}]},"master_zonegroup":"test-zonegroup","master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","realm_name":"test-realm","realm_epoch":4} 2021-02-11T00:43:23.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:23.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:23.782 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:43:23.868 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:23.868 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:23.927 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:43:23.924+0000 7ff0b2689080 0 zone id 7ef30163-aa7c-4a91-ad66-1f645b4c0479 is not a part of zonegroup test-zonegroup 2021-02-11T00:43:23.928 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:43:23.928+0000 7ff0b2689080 0 zone id 7ef30163-aa7c-4a91-ad66-1f645b4c0479 is not a part of zonegroup default 2021-02-11T00:43:23.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:23.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:23.944 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:43:24.024 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:24.024 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:24.083 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T00:43:24.080+0000 7f13a6c8b080 0 ERROR: zonegroup remove_zg has a non existent master zone 2021-02-11T00:43:24.092 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-02-11T00:43:24.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:24.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:24.092 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:43:24.170 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:24.170 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:24.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:24.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-02-11T00:43:24.209 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-02-11T00:43:24.293 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:24.293 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:24.619 INFO:teuthology.orchestra.run.smithi057.stdout:{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","epoch":9,"predecessor_uuid":"4d4a6517-ce7a-42dd-9c62-65c684a638fb","sync_status":["","1_1613003825.727943_198.1","",""],"period_map":{"id":"702c963a-2caf-4007-8190-7df3d99bc8ac","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","zones":[{"id":"35e954f9-5ea0-411e-8e41-e821d2787b31","name":"test-zone3","endpoints":["http://smithi057.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","name":"test-zone4","endpoints":["http://smithi095.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","name":"test-zone1","endpoints":["http://smithi057.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","name":"test-zone2","endpoints":["http://smithi095.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"35e954f9-5ea0-411e-8e41-e821d2787b31","val":398828966},{"key":"d88fea79-91d3-40e7-8bb7-515bb1e5c3b3","val":1248431312},{"key":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","val":3133685257},{"key":"fd9c3d9f-edaa-4c82-8d3a-10c80d5a8605","val":1241439546}]},"master_zonegroup":"test-zonegroup","master_zone":"e989c9f2-9c89-43f5-a54f-e1d0705a9abe","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c192af4-f67d-481a-af12-4cc9f791da82","realm_name":"test-realm","realm_epoch":4} 2021-02-11T00:43:24.619 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2021-02-11T00:43:24.620 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-50 2021-02-11T00:43:28.377 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-51 2021-02-11T00:43:28.544 INFO:tasks.rgw.c2.client.1.smithi095.stdout:*** Caught signal (Segmentation fault) ** 2021-02-11T00:43:28.544 INFO:tasks.rgw.c2.client.1.smithi095.stdout: in thread 7f22effef700 thread_name:rados_async 2021-02-11T00:43:28.549 INFO:tasks.rgw.c2.client.1.smithi095.stdout: ceph version 17.0.0-691-g597c1051 (597c1051df9f309aa7d6c6db46dcbc2c3982c6e3) quincy (dev) 2021-02-11T00:43:28.549 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f2335f48980] 2021-02-11T00:43:28.549 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 2: (RGWSI_SysObj_Core::get_system_obj_state_impl(RGWSysObjectCtxBase*, rgw_raw_obj const&, RGWSysObjState**, RGWObjVersionTracker*, optional_yield, DoutPrefixProvider const*)+0xf39) [0x7f23366fc6a9] 2021-02-11T00:43:28.549 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 3: (RGWSI_SysObj_Core::get_system_obj_state(RGWSysObjectCtxBase*, rgw_raw_obj const&, RGWSysObjState**, RGWObjVersionTracker*, optional_yield, DoutPrefixProvider const*)+0x4f) [0x7f23366fc96f] 2021-02-11T00:43:28.549 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 4: (RGWSI_SysObj_Core::stat(RGWSysObjectCtxBase&, RGWSI_SysObj_Obj_GetObjState&, rgw_raw_obj const&, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, bool, std::chrono::time_point > >*, unsigned long*, RGWObjVersionTracker*, optional_yield, DoutPrefixProvider const*)+0x74) [0x7f23366fca04] 2021-02-11T00:43:28.549 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 5: (RGWSI_SysObj::Obj::ROp::stat(optional_yield, DoutPrefixProvider const*)+0x61) [0x7f23366f8b21] 2021-02-11T00:43:28.549 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 6: (rgw_get_system_obj(RGWSysObjectCtx&, rgw_pool const&, std::__cxx11::basic_string, std::allocator > const&, ceph::buffer::v15_2_0::list&, RGWObjVersionTracker*, std::chrono::time_point > >*, optional_yield, DoutPrefixProvider const*, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, rgw_cache_entry_info*, boost::optional)+0x169) [0x7f2336ad95b9] 2021-02-11T00:43:28.549 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 7: (RGWSI_MetaBackend_SObj::get_entry(RGWSI_MetaBackend::Context*, std::__cxx11::basic_string, std::allocator > const&, RGWSI_MetaBackend::GetParams&, RGWObjVersionTracker*, optional_yield, DoutPrefixProvider const*)+0xec) [0x7f2336bed6dc] 2021-02-11T00:43:28.549 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 8: (RGWSI_User_RADOS::read_user_info(RGWSI_MetaBackend::Context*, rgw_user const&, RGWUserInfo*, RGWObjVersionTracker*, std::chrono::time_point > >*, rgw_cache_entry_info*, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, optional_yield, DoutPrefixProvider const*)+0x2b2) [0x7f2336c09362] 2021-02-11T00:43:28.549 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 9: /usr/lib/libradosgw.so.2(+0x98db78) [0x7f2336ae2b78] 2021-02-11T00:43:28.550 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 10: /usr/lib/libradosgw.so.2(+0xa9608a) [0x7f2336beb08a] 2021-02-11T00:43:28.550 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 11: (RGWSI_MetaBackend_SObj::call(std::optional >, std::function)+0x93) [0x7f2336bee923] 2021-02-11T00:43:28.550 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 12: (RGWSI_MetaBackend_Handler::call(std::optional >, std::function)+0x61) [0x7f2336beb331] 2021-02-11T00:43:28.550 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 13: (RGWSI_MetaBackend_Handler::call(std::function)+0x70) [0x7f2336af30d0] 2021-02-11T00:43:28.550 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 14: (RGWUserCtl::get_info_by_uid(DoutPrefixProvider const*, rgw_user const&, RGWUserInfo*, optional_yield, RGWUserCtl::GetParams const&)+0xa3) [0x7f2336ae3c93] 2021-02-11T00:43:28.550 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 15: (rgw_get_user_info_by_uid(DoutPrefixProvider const*, RGWUserCtl*, rgw_user const&, RGWUserInfo&, optional_yield, RGWObjVersionTracker*, std::chrono::time_point > >*, rgw_cache_entry_info*, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*)+0x50) [0x7f2336ae3d70] 2021-02-11T00:43:28.550 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 16: (RGWUser::init(DoutPrefixProvider const*, RGWUserAdminOpState&, optional_yield)+0x6fd) [0x7f2336aec63d] 2021-02-11T00:43:28.550 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 17: (RGWUser::init(DoutPrefixProvider const*, rgw::sal::RGWRadosStore*, RGWUserAdminOpState&, optional_yield)+0x48) [0x7f2336aec8e8] 2021-02-11T00:43:28.550 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 18: (RGWUserAdminOp_User::create(DoutPrefixProvider const*, rgw::sal::RGWRadosStore*, RGWUserAdminOpState&, RGWFormatterFlusher&, optional_yield)+0x70) [0x7f2336aefff0] 2021-02-11T00:43:28.551 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 19: (RGWSimpleWriteOnlyAsyncCR::Request::_send_request()+0xee8) [0x7f2336957c48] 2021-02-11T00:43:28.551 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 20: (RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x20) [0x7f2336947040] 2021-02-11T00:43:28.551 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 21: (RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0xd) [0x7f233694fbcd] 2021-02-11T00:43:28.551 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 22: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa52) [0x7f232bcb0db2] 2021-02-11T00:43:28.551 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 23: (ThreadPool::WorkThread::entry()+0x11) [0x7f232bcb1ca1] 2021-02-11T00:43:28.551 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 24: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f2335f3d6db] 2021-02-11T00:43:28.551 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 25: clone() 2021-02-11T00:43:28.552 INFO:tasks.rgw.c2.client.1.smithi095.stdout:2021-02-11T00:43:28.546+0000 7f22effef700 -1 *** Caught signal (Segmentation fault) ** 2021-02-11T00:43:28.552 INFO:tasks.rgw.c2.client.1.smithi095.stdout: in thread 7f22effef700 thread_name:rados_async 2021-02-11T00:43:28.552 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 2021-02-11T00:43:28.552 INFO:tasks.rgw.c2.client.1.smithi095.stdout: ceph version 17.0.0-691-g597c1051 (597c1051df9f309aa7d6c6db46dcbc2c3982c6e3) quincy (dev) 2021-02-11T00:43:28.552 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f2335f48980] 2021-02-11T00:43:28.552 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 2: (RGWSI_SysObj_Core::get_system_obj_state_impl(RGWSysObjectCtxBase*, rgw_raw_obj const&, RGWSysObjState**, RGWObjVersionTracker*, optional_yield, DoutPrefixProvider const*)+0xf39) [0x7f23366fc6a9] 2021-02-11T00:43:28.552 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 3: (RGWSI_SysObj_Core::get_system_obj_state(RGWSysObjectCtxBase*, rgw_raw_obj const&, RGWSysObjState**, RGWObjVersionTracker*, optional_yield, DoutPrefixProvider const*)+0x4f) [0x7f23366fc96f] 2021-02-11T00:43:28.552 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 4: (RGWSI_SysObj_Core::stat(RGWSysObjectCtxBase&, RGWSI_SysObj_Obj_GetObjState&, rgw_raw_obj const&, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, bool, std::chrono::time_point > >*, unsigned long*, RGWObjVersionTracker*, optional_yield, DoutPrefixProvider const*)+0x74) [0x7f23366fca04] 2021-02-11T00:43:28.552 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 5: (RGWSI_SysObj::Obj::ROp::stat(optional_yield, DoutPrefixProvider const*)+0x61) [0x7f23366f8b21] 2021-02-11T00:43:28.552 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 6: (rgw_get_system_obj(RGWSysObjectCtx&, rgw_pool const&, std::__cxx11::basic_string, std::allocator > const&, ceph::buffer::v15_2_0::list&, RGWObjVersionTracker*, std::chrono::time_point > >*, optional_yield, DoutPrefixProvider const*, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, rgw_cache_entry_info*, boost::optional)+0x169) [0x7f2336ad95b9] 2021-02-11T00:43:28.552 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 7: (RGWSI_MetaBackend_SObj::get_entry(RGWSI_MetaBackend::Context*, std::__cxx11::basic_string, std::allocator > const&, RGWSI_MetaBackend::GetParams&, RGWObjVersionTracker*, optional_yield, DoutPrefixProvider const*)+0xec) [0x7f2336bed6dc] 2021-02-11T00:43:28.553 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 8: (RGWSI_User_RADOS::read_user_info(RGWSI_MetaBackend::Context*, rgw_user const&, RGWUserInfo*, RGWObjVersionTracker*, std::chrono::time_point > >*, rgw_cache_entry_info*, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, optional_yield, DoutPrefixProvider const*)+0x2b2) [0x7f2336c09362] 2021-02-11T00:43:28.553 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 9: /usr/lib/libradosgw.so.2(+0x98db78) [0x7f2336ae2b78] 2021-02-11T00:43:28.553 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 10: /usr/lib/libradosgw.so.2(+0xa9608a) [0x7f2336beb08a] 2021-02-11T00:43:28.553 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 11: (RGWSI_MetaBackend_SObj::call(std::optional >, std::function)+0x93) [0x7f2336bee923] 2021-02-11T00:43:28.553 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 12: (RGWSI_MetaBackend_Handler::call(std::optional >, std::function)+0x61) [0x7f2336beb331] 2021-02-11T00:43:28.553 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 13: (RGWSI_MetaBackend_Handler::call(std::function)+0x70) [0x7f2336af30d0] 2021-02-11T00:43:28.553 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 14: (RGWUserCtl::get_info_by_uid(DoutPrefixProvider const*, rgw_user const&, RGWUserInfo*, optional_yield, RGWUserCtl::GetParams const&)+0xa3) [0x7f2336ae3c93] 2021-02-11T00:43:28.553 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 15: (rgw_get_user_info_by_uid(DoutPrefixProvider const*, RGWUserCtl*, rgw_user const&, RGWUserInfo&, optional_yield, RGWObjVersionTracker*, std::chrono::time_point > >*, rgw_cache_entry_info*, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*)+0x50) [0x7f2336ae3d70] 2021-02-11T00:43:28.553 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 16: (RGWUser::init(DoutPrefixProvider const*, RGWUserAdminOpState&, optional_yield)+0x6fd) [0x7f2336aec63d] 2021-02-11T00:43:28.553 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 17: (RGWUser::init(DoutPrefixProvider const*, rgw::sal::RGWRadosStore*, RGWUserAdminOpState&, optional_yield)+0x48) [0x7f2336aec8e8] 2021-02-11T00:43:28.553 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 18: (RGWUserAdminOp_User::create(DoutPrefixProvider const*, rgw::sal::RGWRadosStore*, RGWUserAdminOpState&, RGWFormatterFlusher&, optional_yield)+0x70) [0x7f2336aefff0] 2021-02-11T00:43:28.553 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 19: (RGWSimpleWriteOnlyAsyncCR::Request::_send_request()+0xee8) [0x7f2336957c48] 2021-02-11T00:43:28.554 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 20: (RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x20) [0x7f2336947040] 2021-02-11T00:43:28.554 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 21: (RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0xd) [0x7f233694fbcd] 2021-02-11T00:43:28.554 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 22: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa52) [0x7f232bcb0db2] 2021-02-11T00:43:28.554 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 23: (ThreadPool::WorkThread::entry()+0x11) [0x7f232bcb1ca1] 2021-02-11T00:43:28.554 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 24: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f2335f3d6db] 2021-02-11T00:43:28.554 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 25: clone() 2021-02-11T00:43:28.554 INFO:tasks.rgw.c2.client.1.smithi095.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-02-11T00:43:28.554 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 2021-02-11T00:43:28.569 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 0> 2021-02-11T00:43:28.546+0000 7f22effef700 -1 *** Caught signal (Segmentation fault) ** 2021-02-11T00:43:28.569 INFO:tasks.rgw.c2.client.1.smithi095.stdout: in thread 7f22effef700 thread_name:rados_async 2021-02-11T00:43:28.569 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 2021-02-11T00:43:28.569 INFO:tasks.rgw.c2.client.1.smithi095.stdout: ceph version 17.0.0-691-g597c1051 (597c1051df9f309aa7d6c6db46dcbc2c3982c6e3) quincy (dev) 2021-02-11T00:43:28.569 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f2335f48980] 2021-02-11T00:43:28.569 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 2: (RGWSI_SysObj_Core::get_system_obj_state_impl(RGWSysObjectCtxBase*, rgw_raw_obj const&, RGWSysObjState**, RGWObjVersionTracker*, optional_yield, DoutPrefixProvider const*)+0xf39) [0x7f23366fc6a9] 2021-02-11T00:43:28.570 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 3: (RGWSI_SysObj_Core::get_system_obj_state(RGWSysObjectCtxBase*, rgw_raw_obj const&, RGWSysObjState**, RGWObjVersionTracker*, optional_yield, DoutPrefixProvider const*)+0x4f) [0x7f23366fc96f] 2021-02-11T00:43:28.570 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 4: (RGWSI_SysObj_Core::stat(RGWSysObjectCtxBase&, RGWSI_SysObj_Obj_GetObjState&, rgw_raw_obj const&, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, bool, std::chrono::time_point > >*, unsigned long*, RGWObjVersionTracker*, optional_yield, DoutPrefixProvider const*)+0x74) [0x7f23366fca04] 2021-02-11T00:43:28.570 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 5: (RGWSI_SysObj::Obj::ROp::stat(optional_yield, DoutPrefixProvider const*)+0x61) [0x7f23366f8b21] 2021-02-11T00:43:28.570 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 6: (rgw_get_system_obj(RGWSysObjectCtx&, rgw_pool const&, std::__cxx11::basic_string, std::allocator > const&, ceph::buffer::v15_2_0::list&, RGWObjVersionTracker*, std::chrono::time_point > >*, optional_yield, DoutPrefixProvider const*, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, rgw_cache_entry_info*, boost::optional)+0x169) [0x7f2336ad95b9] 2021-02-11T00:43:28.570 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 7: (RGWSI_MetaBackend_SObj::get_entry(RGWSI_MetaBackend::Context*, std::__cxx11::basic_string, std::allocator > const&, RGWSI_MetaBackend::GetParams&, RGWObjVersionTracker*, optional_yield, DoutPrefixProvider const*)+0xec) [0x7f2336bed6dc] 2021-02-11T00:43:28.570 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 8: (RGWSI_User_RADOS::read_user_info(RGWSI_MetaBackend::Context*, rgw_user const&, RGWUserInfo*, RGWObjVersionTracker*, std::chrono::time_point > >*, rgw_cache_entry_info*, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, optional_yield, DoutPrefixProvider const*)+0x2b2) [0x7f2336c09362] 2021-02-11T00:43:28.570 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 9: /usr/lib/libradosgw.so.2(+0x98db78) [0x7f2336ae2b78] 2021-02-11T00:43:28.570 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 10: /usr/lib/libradosgw.so.2(+0xa9608a) [0x7f2336beb08a] 2021-02-11T00:43:28.570 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 11: (RGWSI_MetaBackend_SObj::call(std::optional >, std::function)+0x93) [0x7f2336bee923] 2021-02-11T00:43:28.570 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 12: (RGWSI_MetaBackend_Handler::call(std::optional >, std::function)+0x61) [0x7f2336beb331] 2021-02-11T00:43:28.570 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 13: (RGWSI_MetaBackend_Handler::call(std::function)+0x70) [0x7f2336af30d0] 2021-02-11T00:43:28.570 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 14: (RGWUserCtl::get_info_by_uid(DoutPrefixProvider const*, rgw_user const&, RGWUserInfo*, optional_yield, RGWUserCtl::GetParams const&)+0xa3) [0x7f2336ae3c93] 2021-02-11T00:43:28.571 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 15: (rgw_get_user_info_by_uid(DoutPrefixProvider const*, RGWUserCtl*, rgw_user const&, RGWUserInfo&, optional_yield, RGWObjVersionTracker*, std::chrono::time_point > >*, rgw_cache_entry_info*, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*)+0x50) [0x7f2336ae3d70] 2021-02-11T00:43:28.571 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 16: (RGWUser::init(DoutPrefixProvider const*, RGWUserAdminOpState&, optional_yield)+0x6fd) [0x7f2336aec63d] 2021-02-11T00:43:28.571 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 17: (RGWUser::init(DoutPrefixProvider const*, rgw::sal::RGWRadosStore*, RGWUserAdminOpState&, optional_yield)+0x48) [0x7f2336aec8e8] 2021-02-11T00:43:28.571 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 18: (RGWUserAdminOp_User::create(DoutPrefixProvider const*, rgw::sal::RGWRadosStore*, RGWUserAdminOpState&, RGWFormatterFlusher&, optional_yield)+0x70) [0x7f2336aefff0] 2021-02-11T00:43:28.571 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 19: (RGWSimpleWriteOnlyAsyncCR::Request::_send_request()+0xee8) [0x7f2336957c48] 2021-02-11T00:43:28.571 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 20: (RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x20) [0x7f2336947040] 2021-02-11T00:43:28.571 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 21: (RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0xd) [0x7f233694fbcd] 2021-02-11T00:43:28.571 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 22: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa52) [0x7f232bcb0db2] 2021-02-11T00:43:28.571 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 23: (ThreadPool::WorkThread::entry()+0x11) [0x7f232bcb1ca1] 2021-02-11T00:43:28.571 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 24: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f2335f3d6db] 2021-02-11T00:43:28.571 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 25: clone() 2021-02-11T00:43:28.571 INFO:tasks.rgw.c2.client.1.smithi095.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-02-11T00:43:28.571 INFO:tasks.rgw.c2.client.1.smithi095.stdout: 2021-02-11T00:43:28.589 INFO:tasks.rgw.c2.client.1.smithi095.stderr:daemon-helper: command crashed with signal 11 2021-02-11T00:43:33.474 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-52 2021-02-11T00:43:33.549 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-53 2021-02-11T00:43:33.581 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-54 2021-02-11T00:43:33.643 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-55 2021-02-11T00:43:33.938 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-02-11T00:43:35.906 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2021-02-11T00:43:35.907 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-56 2021-02-11T00:43:35.940 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-57 2021-02-11T00:43:36.010 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-58 2021-02-11T00:43:36.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:36.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:36.069 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:43:36.103 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:36.103 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:36.331 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T00:43:36.373 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613004216.047042_417.1","last_update":"2021-02-11T00:43:36.047042Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:43:36.373 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:43:36.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:36.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:36.374 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:43:36.413 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:36.414 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:36.665 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:43:36.666 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:43:39.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~6s 2021-02-11T00:43:41.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:41.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:41.667 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:43:41.708 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:41.708 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:41.969 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:43:41.970 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:43:45.151 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~11s 2021-02-11T00:43:46.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:46.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:46.971 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:43:47.010 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:47.051 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:47.263 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:43:47.263 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:43:50.759 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~17s 2021-02-11T00:43:52.264 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:52.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:52.318 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:43:52.358 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:52.358 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:52.619 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:43:52.619 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:43:56.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~22s 2021-02-11T00:43:57.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:57.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:43:57.621 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:43:57.666 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:43:57.666 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:43:57.916 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:43:57.916 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:44:01.972 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~28s 2021-02-11T00:44:02.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:02.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:02.917 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:44:02.959 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:44:02.959 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:44:03.225 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:44:03.225 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:44:07.579 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~34s 2021-02-11T00:44:08.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:08.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:08.226 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:44:08.270 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:44:08.270 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:44:08.510 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:44:08.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:44:13.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~39s 2021-02-11T00:44:13.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:13.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:13.512 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:44:13.553 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:44:13.553 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:44:13.810 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:44:13.810 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:44:18.792 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~45s 2021-02-11T00:44:18.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:18.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:18.812 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:44:18.853 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:44:18.853 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:44:19.105 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:44:19.105 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:44:24.106 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:24.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:24.107 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:44:24.142 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:44:24.142 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:44:24.385 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:44:24.385 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:44:24.398 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~50s 2021-02-11T00:44:29.387 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:29.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:29.387 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:44:29.429 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:44:29.429 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:44:29.693 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:44:29.693 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:44:30.003 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~56s 2021-02-11T00:44:34.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:34.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:34.699 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:44:34.741 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:44:34.741 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:44:34.989 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:44:34.989 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:44:35.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~62s 2021-02-11T00:44:39.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:39.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:39.991 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:44:40.030 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:44:40.030 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:44:40.264 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:44:40.264 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:44:41.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~67s 2021-02-11T00:44:45.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:45.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:45.265 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:44:45.307 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:44:45.307 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:44:45.559 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:44:45.560 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:44:46.822 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~73s 2021-02-11T00:44:50.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:50.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:50.562 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:44:50.602 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:44:50.602 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:44:50.857 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:44:50.858 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:44:52.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~79s 2021-02-11T00:44:55.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:55.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:44:55.860 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:44:55.901 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:44:55.901 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:44:56.140 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:44:56.140 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:44:58.258 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~84s 2021-02-11T00:45:01.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:01.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:01.142 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:45:01.185 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:45:01.185 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:45:01.451 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:45:01.452 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:45:03.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~90s 2021-02-11T00:45:06.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:06.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:06.453 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:45:06.494 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:45:06.494 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:45:06.752 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:45:06.752 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:45:09.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~96s 2021-02-11T00:45:11.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:11.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:11.754 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:45:11.795 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:45:11.795 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:45:12.054 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:45:12.055 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:45:15.077 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~101s 2021-02-11T00:45:17.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:17.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:17.056 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:45:17.096 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:45:17.096 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:45:17.341 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:45:17.342 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:45:20.683 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~107s 2021-02-11T00:45:22.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:22.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:22.343 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:45:22.386 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:45:22.387 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:45:22.645 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613004213.513737_385.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:43:33.513737Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:45:22.645 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613004213.513737_385.1 2021-02-11T00:45:26.289 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~112s 2021-02-11T00:45:27.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:27.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:27.646 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:45:27.688 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:45:27.688 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:45:27.944 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613004213.513737_385.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:43:33.513737Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:45:27.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613004213.513737_385.1 2021-02-11T00:45:31.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~118s 2021-02-11T00:45:32.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:32.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:32.946 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:45:32.989 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:45:32.990 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:45:33.228 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613004213.513737_385.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:43:33.513737Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:45:33.228 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613004213.513737_385.1 2021-02-11T00:45:37.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~124s 2021-02-11T00:45:38.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:38.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:38.230 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:45:38.270 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:45:38.271 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:45:38.525 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613004216.047042_417.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:43:36.047042Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:45:38.526 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:45:38.526 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:45:38.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:38.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:38.527 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:45:38.567 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:45:38.568 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:45:38.812 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613004216.047042_417.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:43:36.047042Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:45:38.812 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:45:38.812 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:45:38.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:38.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:38.813 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:45:38.853 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:45:38.853 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:45:39.104 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:45:39.104 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:45:43.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~129s 2021-02-11T00:45:44.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:44.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:44.288 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:45:44.321 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:45:44.321 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:45:44.551 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:45:44.552 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:45:49.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:49.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:49.554 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:45:49.595 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:45:49.595 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:45:49.840 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:45:49.841 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:45:49.891 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~136s 2021-02-11T00:45:54.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:54.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:45:54.842 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:45:54.883 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:45:54.883 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:45:55.119 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:45:55.119 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:45:55.496 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~142s 2021-02-11T00:46:00.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:00.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:00.121 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:46:00.155 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:46:00.155 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:46:00.399 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:46:00.399 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:46:01.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~147s 2021-02-11T00:46:05.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:05.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:05.401 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:46:05.443 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:46:05.443 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:46:05.704 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:46:05.704 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:46:06.710 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~153s 2021-02-11T00:46:10.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:10.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:10.706 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:46:10.770 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:46:10.771 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:46:11.018 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:46:11.019 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:46:12.316 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~158s 2021-02-11T00:46:16.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:16.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:16.020 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:46:16.062 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:46:16.062 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:46:16.280 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:46:16.280 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:46:17.923 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~164s 2021-02-11T00:46:21.281 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:21.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:21.282 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:46:21.321 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:46:21.321 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:46:21.557 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:46:21.558 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:46:23.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~170s 2021-02-11T00:46:26.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:26.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:26.559 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:46:26.600 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:46:26.600 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:46:26.851 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:46:26.851 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:46:29.134 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~175s 2021-02-11T00:46:31.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:31.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:31.853 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:46:31.893 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:46:31.894 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:46:32.130 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:46:32.131 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:46:34.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~181s 2021-02-11T00:46:37.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:37.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:37.132 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:46:37.173 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:46:37.173 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:46:37.416 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:46:37.416 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:46:40.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~186s 2021-02-11T00:46:42.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:42.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:42.418 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:46:42.458 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:46:42.458 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:46:42.710 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:46:42.711 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:46:45.949 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~192s 2021-02-11T00:46:47.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:47.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:47.712 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:46:47.754 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:46:47.755 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:46:48.006 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:46:48.006 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:46:51.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~198s 2021-02-11T00:46:53.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:53.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:53.008 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:46:53.052 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:46:53.052 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:46:53.291 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:46:53.292 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:46:57.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~203s 2021-02-11T00:46:58.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:58.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:46:58.294 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:46:58.335 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:46:58.335 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:46:58.574 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:46:58.574 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:47:02.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~209s 2021-02-11T00:47:03.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:03.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:03.576 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:47:03.619 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:47:03.619 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:47:03.869 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:47:03.870 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:47:08.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~214s 2021-02-11T00:47:08.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:08.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:08.872 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:47:08.913 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:47:08.913 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:47:09.140 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:47:09.141 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:47:13.970 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~220s 2021-02-11T00:47:14.142 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:14.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:14.142 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:47:14.184 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:47:14.184 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:47:14.428 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:47:14.428 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:47:19.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:19.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:19.430 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:47:19.470 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:47:19.470 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:47:19.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~226s 2021-02-11T00:47:19.718 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:47:19.719 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:47:24.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:24.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:24.720 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:47:24.761 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:47:24.761 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:47:25.017 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:47:25.017 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:47:25.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~231s 2021-02-11T00:47:30.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:30.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:30.019 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:47:30.058 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:47:30.058 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:47:30.313 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:47:30.313 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:47:30.785 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~237s 2021-02-11T00:47:35.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:35.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:35.314 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:47:35.355 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:47:35.355 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:47:35.615 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:47:35.615 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:47:36.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~242s 2021-02-11T00:47:40.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:40.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:40.617 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:47:40.657 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:47:40.657 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:47:40.907 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:47:40.907 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:47:41.997 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~248s 2021-02-11T00:47:45.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:45.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:45.909 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:47:45.949 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:47:45.949 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:47:46.209 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:47:46.209 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:47:47.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~254s 2021-02-11T00:47:51.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:51.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:51.211 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:47:51.252 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:47:51.252 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:47:51.491 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:47:51.492 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:47:53.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~259s 2021-02-11T00:47:56.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:56.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:47:56.493 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:47:56.533 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:47:56.533 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:47:56.781 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:47:56.781 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:47:58.818 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~265s 2021-02-11T00:48:01.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:01.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:01.783 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:48:01.822 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:48:01.822 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:48:02.074 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:48:02.075 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:48:04.424 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~270s 2021-02-11T00:48:07.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:07.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:07.077 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:48:07.120 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:48:07.120 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:48:07.367 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:48:07.367 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:48:10.031 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~276s 2021-02-11T00:48:12.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:12.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:12.368 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:48:12.409 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:48:12.409 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:48:12.680 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:48:12.680 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:48:15.639 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~282s 2021-02-11T00:48:17.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:17.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:17.682 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:48:17.723 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:48:17.723 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:48:17.977 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:48:17.977 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:48:21.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~287s 2021-02-11T00:48:22.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:22.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:22.978 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:48:23.017 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:48:23.018 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:48:23.275 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:48:23.275 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:48:26.851 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~293s 2021-02-11T00:48:28.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:28.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:28.277 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:48:28.317 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:48:28.317 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:48:28.571 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:48:28.571 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:48:32.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~299s 2021-02-11T00:48:33.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:33.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:33.572 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:48:33.614 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:48:33.615 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:48:33.861 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:48:33.862 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:48:38.065 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~304s 2021-02-11T00:48:38.066 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2021-02-11T00:48:38.066 ERROR:tasks.daemonwatchdog.daemon_watchdog:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/daemonwatchdog.py", line 37, in _run self.watch() File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/daemonwatchdog.py", line 113, in watch self.bark() File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/daemonwatchdog.py", line 53, in bark for mount in self.ctx.mounts.values(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/config.py", line 255, in __getattr__ raise AttributeError(name) AttributeError: mounts 2021-02-11T00:48:38.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:38.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:38.863 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:48:38.906 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:48:38.906 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:48:39.155 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:48:39.156 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:48:44.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:44.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:44.158 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:48:44.198 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:48:44.199 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:48:44.457 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:48:44.458 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:48:49.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:49.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:49.459 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:48:49.498 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:48:49.499 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:48:49.740 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:48:49.740 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:48:54.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:54.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:48:54.742 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:48:54.782 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:48:54.782 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:48:55.024 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:48:55.025 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:49:00.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:00.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:00.026 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:49:00.065 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:49:00.065 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:49:00.294 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:49:00.295 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:49:05.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:05.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:05.297 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:49:05.337 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:49:05.337 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:49:05.593 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:49:05.593 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:49:10.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:10.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:10.594 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:49:10.636 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:49:10.636 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:49:10.890 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:49:10.890 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:49:15.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:15.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:15.892 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:49:15.934 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:49:15.934 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:49:16.187 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:49:16.187 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:49:21.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:21.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:21.189 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:49:21.230 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:49:21.230 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:49:21.477 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:49:21.478 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:49:26.478 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:26.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:26.479 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:49:26.522 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:49:26.522 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:49:26.769 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:49:26.770 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:49:31.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:31.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:31.772 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:49:31.814 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:49:31.814 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:49:32.056 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:49:32.056 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:49:37.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:37.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:37.058 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:49:37.098 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:49:37.098 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:49:37.347 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:49:37.348 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:49:42.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:42.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:42.349 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:49:42.389 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:49:42.389 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:49:42.639 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:49:42.639 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:49:47.640 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:47.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:47.641 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:49:47.681 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:49:47.681 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:49:47.947 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:49:47.948 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:49:52.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:52.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:52.949 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:49:52.989 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:49:52.990 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:49:53.224 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:49:53.225 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:49:58.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:58.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:49:58.226 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:49:58.264 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:49:58.264 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:49:58.510 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:49:58.510 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:50:03.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:03.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:03.512 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:50:03.553 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:50:03.553 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:50:03.792 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:50:03.793 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:50:08.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:08.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:08.794 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:50:08.834 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:50:08.834 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:50:09.087 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:50:09.088 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:50:14.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:14.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:14.089 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:50:14.131 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:50:14.131 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:50:14.381 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:50:14.381 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:50:19.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:19.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:19.382 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:50:19.424 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:50:19.425 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:50:19.677 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:50:19.678 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:50:24.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:24.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:24.679 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:50:24.722 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:50:24.722 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:50:24.973 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:50:24.974 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:50:29.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:29.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:29.975 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:50:30.018 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:50:30.018 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:50:30.265 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:50:30.265 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:50:35.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:35.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:35.266 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:50:35.305 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:50:35.305 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:50:35.548 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:50:35.548 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:50:40.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:40.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:40.550 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:50:40.591 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:50:40.591 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:50:40.842 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:50:40.842 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:50:45.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:45.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:45.844 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:50:45.886 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:50:45.886 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:50:46.151 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:50:46.222 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:50:51.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:51.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:51.223 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:50:51.261 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:50:51.261 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:50:51.516 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:50:51.516 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004216.047042_417.1 target=1_1613003977.863186_285.1 2021-02-11T00:50:56.542 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... FAIL 2021-02-11T00:50:56.543 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-59 2021-02-11T00:50:56.592 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-60 2021-02-11T00:50:56.673 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-61 2021-02-11T00:50:57.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:57.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:57.307 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:50:57.353 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:50:57.353 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:50:57.532 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T00:50:57.573 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613004656.710510_430.1","last_update":"2021-02-11T00:50:56.710510Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:50:57.573 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:50:57.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:57.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:57.573 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:50:57.613 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:50:57.613 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:50:57.857 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613004656.710510_430.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:50:56.710510Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:50:57.857 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:50:57.857 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:50:57.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:57.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:57.858 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:50:57.898 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:50:57.898 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:50:58.103 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613004656.710510_430.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:50:56.710510Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:50:58.103 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:50:58.104 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:50:58.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:58.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:50:58.104 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:50:58.145 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:50:58.145 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:50:58.404 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:50:58.405 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:51:03.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:03.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:03.407 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:51:03.446 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:51:03.446 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:51:03.691 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:51:03.692 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:51:08.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:08.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:08.693 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:51:08.733 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:51:08.733 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:51:08.983 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:51:08.983 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:51:13.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:13.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:13.985 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:51:14.026 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:51:14.026 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:51:14.284 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:51:14.284 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:51:19.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:19.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:19.286 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:51:19.327 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:51:19.328 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:51:19.588 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:51:19.588 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:51:24.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:24.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:24.589 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:51:24.630 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:51:24.630 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:51:24.895 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:51:24.895 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:51:29.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:29.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:29.897 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:51:29.940 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:51:29.940 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:51:30.185 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:51:30.185 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:51:35.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:35.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:35.187 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:51:35.228 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:51:35.228 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:51:35.474 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:51:35.474 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:51:40.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:40.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:40.476 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:51:40.517 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:51:40.517 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:51:40.771 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:51:40.771 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:51:45.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:45.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:45.773 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:51:45.814 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:51:45.814 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:51:46.061 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:51:46.061 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:51:51.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:51.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:51.063 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:51:51.107 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:51:51.108 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:51:51.361 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:51:51.361 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:51:56.362 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:56.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:51:56.363 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:51:56.404 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:51:56.404 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:51:56.665 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:51:56.665 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:52:01.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:01.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:01.666 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:52:01.705 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:52:01.705 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:52:01.979 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:52:01.980 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:52:06.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:06.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:06.981 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:52:07.021 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:52:07.021 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:52:07.276 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:52:07.276 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:52:12.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:12.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:12.277 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:52:12.317 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:52:12.317 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:52:12.586 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:52:12.586 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:52:17.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:17.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:17.588 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:52:17.630 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:52:17.630 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:52:17.876 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:52:17.877 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:52:22.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:22.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:22.878 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:52:22.921 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:52:22.922 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:52:23.168 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:52:23.169 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:52:28.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:28.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:28.171 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:52:28.211 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:52:28.211 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:52:28.476 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:52:28.476 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:52:33.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:33.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:33.478 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:52:33.521 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:52:33.521 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:52:33.779 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:52:33.780 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:52:38.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:38.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:38.781 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:52:38.820 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:52:38.821 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:52:39.065 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:52:39.065 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:52:44.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:44.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:44.067 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:52:44.111 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:52:44.111 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:52:44.372 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:52:44.372 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:52:49.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:49.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:49.374 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:52:49.413 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:52:49.414 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:52:49.680 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:52:49.681 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:52:54.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:54.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:54.682 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:52:54.721 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:52:54.721 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:52:54.968 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:52:54.968 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:52:59.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:59.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:52:59.970 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:53:00.011 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:53:00.011 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:53:00.238 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:53:00.239 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:53:05.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:05.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:05.241 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:53:05.281 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:53:05.281 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:53:05.543 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:53:05.544 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:53:10.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:10.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:10.545 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:53:10.586 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:53:10.586 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:53:10.852 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:53:10.852 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:53:15.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:15.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:15.854 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:53:15.893 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:53:15.893 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:53:16.132 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:53:16.132 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:53:21.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:21.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:21.134 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:53:21.168 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:53:21.168 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:53:21.410 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:53:21.411 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:53:26.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:26.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:26.412 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:53:26.452 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:53:26.452 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:53:26.715 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:53:26.715 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:53:31.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:31.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:31.716 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:53:31.758 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:53:31.758 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:53:32.015 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:53:32.016 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:53:37.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:37.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:37.017 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:53:37.060 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:53:37.060 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:53:37.320 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:53:37.321 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:53:42.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:42.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:42.322 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:53:42.362 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:53:42.363 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:53:42.614 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:53:42.614 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:53:47.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:47.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:47.615 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:53:47.656 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:53:47.656 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:53:47.907 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:53:47.907 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:53:52.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:52.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:52.909 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:53:52.951 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:53:52.951 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:53:53.180 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:53:53.180 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:53:58.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:58.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:53:58.182 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:53:58.223 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:53:58.223 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:53:58.470 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:53:58.470 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:54:03.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:03.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:03.472 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:54:03.512 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:54:03.512 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:54:03.764 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:54:03.765 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:54:08.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:08.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:08.766 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:54:08.807 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:54:08.807 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:54:09.054 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:54:09.055 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:54:14.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:14.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:14.056 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:54:14.097 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:54:14.097 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:54:14.335 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:54:14.336 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:54:19.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:19.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:19.337 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:54:19.380 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:54:19.380 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:54:19.635 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:54:19.635 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:54:24.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:24.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:24.637 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:54:24.679 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:54:24.679 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:54:24.939 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:54:24.939 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:54:29.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:29.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:29.941 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:54:29.983 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:54:29.983 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:54:30.237 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:54:30.237 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:54:35.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:35.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:35.663 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:54:35.705 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:54:35.706 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:54:35.972 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:54:35.972 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:54:40.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:40.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:40.974 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:54:41.014 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:54:41.014 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:54:41.242 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:54:41.243 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:54:46.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:46.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:46.244 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:54:46.281 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:54:46.281 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:54:46.539 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:54:46.540 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:54:51.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:51.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:51.541 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:54:51.580 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:54:51.580 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:54:51.837 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:54:51.838 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:54:56.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:56.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:54:56.839 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:54:56.876 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:54:56.876 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:54:57.127 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:54:57.128 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:55:02.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:02.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:02.130 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:55:02.171 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:55:02.171 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:55:02.423 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:55:02.424 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:55:07.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:07.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:07.425 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:55:07.467 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:55:07.467 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:55:07.726 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:55:07.726 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:55:12.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:12.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:12.728 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:55:12.768 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:55:12.768 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:55:13.025 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:55:13.026 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:55:18.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:18.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:18.028 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:55:18.069 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:55:18.069 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:55:18.320 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:55:18.321 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:55:23.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:23.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:23.322 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:55:23.362 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:55:23.363 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:55:23.610 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:55:23.610 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:55:28.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:28.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:28.611 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:55:28.652 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:55:28.652 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:55:28.918 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:55:28.918 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:55:33.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:33.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:33.920 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:55:33.960 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:55:33.961 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:55:34.217 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:55:34.217 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:55:39.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:39.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:39.219 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:55:39.262 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:55:39.262 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:55:39.506 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:55:39.506 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:55:44.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:44.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:44.508 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:55:44.548 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:55:44.548 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:55:44.821 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:55:44.822 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:55:49.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:49.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:49.823 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:55:49.865 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:55:49.865 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:55:50.107 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:55:50.107 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:55:55.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:55.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:55:55.109 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:55:55.146 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:55:55.146 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:55:55.384 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:55:55.385 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:56:00.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:00.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:00.386 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:56:00.428 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:56:00.429 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:56:00.686 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:56:00.686 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:56:05.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:05.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:05.688 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:56:05.729 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:56:05.730 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:56:05.996 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:56:05.996 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:56:10.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:10.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:10.998 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:56:11.045 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:56:11.046 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:56:11.292 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:56:11.292 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004656.710510_430.1 target=1_1613003977.863186_285.1 2021-02-11T00:56:16.332 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... FAIL 2021-02-11T00:56:16.333 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-62 2021-02-11T00:56:16.380 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-63 2021-02-11T00:56:16.461 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-64 2021-02-11T00:56:17.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:17.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:17.111 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:56:17.153 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:56:17.153 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:56:17.339 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T00:56:17.371 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613004976.507018_460.1","last_update":"2021-02-11T00:56:16.507018Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T00:56:17.372 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T00:56:17.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:17.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:17.372 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:56:17.413 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:56:17.413 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:56:17.653 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613004976.507018_460.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:56:16.507018Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:56:17.653 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T00:56:17.653 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T00:56:17.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:17.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:17.654 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:56:17.694 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:56:17.694 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:56:17.944 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613004976.507018_460.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:56:16.507018Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:56:17.944 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T00:56:17.944 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T00:56:17.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:17.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:17.945 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:56:17.984 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:56:17.984 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:56:18.236 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:56:18.237 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:56:23.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:23.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:23.239 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:56:23.279 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:56:23.279 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:56:23.525 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:56:23.525 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:56:28.526 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:28.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:28.526 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:56:28.569 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:56:28.569 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:56:28.829 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:56:28.829 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:56:33.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:33.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:33.831 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:56:33.873 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:56:33.873 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:56:34.135 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:56:34.135 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:56:39.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:39.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:39.137 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:56:39.173 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:56:39.173 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:56:39.430 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:56:39.431 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:56:44.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:44.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:44.554 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:56:44.595 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:56:44.595 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:56:44.842 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:56:44.843 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:56:49.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:49.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:49.844 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:56:49.885 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:56:49.885 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:56:50.145 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:56:50.145 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:56:55.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:55.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:56:55.147 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:56:55.186 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:56:55.187 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:56:55.438 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:56:55.439 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:57:00.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:00.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:00.440 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:57:00.480 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:57:00.480 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:57:00.739 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:57:00.740 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:57:05.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:06.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:06.212 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:57:06.252 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:57:06.252 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:57:06.505 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:57:06.505 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:57:11.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:11.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:11.507 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:57:11.547 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:57:11.548 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:57:11.811 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:57:11.812 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:57:16.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:16.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:16.814 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:57:16.855 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:57:16.855 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:57:17.104 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:57:17.104 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:57:22.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:22.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:22.106 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:57:22.148 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:57:22.149 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:57:22.392 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:57:22.392 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:57:27.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:27.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:27.393 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:57:27.433 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:57:27.433 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:57:27.683 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:57:27.683 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:57:32.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:32.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:32.686 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:57:32.727 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:57:32.727 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:57:32.980 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:57:32.981 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:57:37.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:37.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:37.983 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:57:38.023 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:57:38.024 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:57:38.274 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:57:38.274 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:57:43.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:43.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:43.276 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:57:43.316 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:57:43.316 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:57:43.583 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:57:43.583 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:57:48.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:48.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:48.584 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:57:48.627 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:57:48.627 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:57:48.888 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:57:48.888 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:57:53.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:53.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:53.890 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:57:53.931 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:57:53.931 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:57:54.195 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:57:54.196 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:57:59.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:59.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:57:59.197 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:57:59.236 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:57:59.236 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:57:59.510 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:57:59.510 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:58:04.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:04.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:04.512 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:58:04.553 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:58:04.554 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:58:04.815 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:58:04.816 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:58:09.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:09.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:09.817 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:58:09.859 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:58:09.859 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:58:10.120 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:58:10.121 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:58:15.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:15.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:15.122 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:58:15.158 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:58:15.158 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:58:15.420 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:58:15.420 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:58:20.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:20.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:20.451 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:58:20.491 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:58:20.492 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:58:20.758 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:58:20.759 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:58:25.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:25.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:25.760 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:58:25.800 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:58:25.800 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:58:26.049 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:58:26.050 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:58:31.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:31.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:31.051 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:58:31.091 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:58:31.092 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:58:31.341 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:58:31.342 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:58:36.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:36.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:36.343 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:58:36.383 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:58:36.383 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:58:36.649 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:58:36.650 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:58:41.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:41.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:41.652 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:58:41.691 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:58:41.692 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:58:41.958 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:58:41.959 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:58:46.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:46.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:46.960 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:58:47.003 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:58:47.003 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:58:47.256 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:58:47.257 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:58:52.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:52.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:52.258 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:58:52.300 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:58:52.300 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:58:52.547 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:58:52.548 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:58:57.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:57.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:58:57.549 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:58:57.591 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:58:57.591 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:58:57.850 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:58:57.851 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:59:02.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:02.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:02.852 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:59:02.894 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:59:02.894 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:59:03.149 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:59:03.150 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:59:08.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:08.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:08.151 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:59:08.190 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:59:08.190 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:59:08.453 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:59:08.453 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:59:13.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:13.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:13.455 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:59:13.497 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:59:13.497 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:59:13.767 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:59:13.767 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:59:18.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:18.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:18.769 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:59:18.810 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:59:18.810 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:59:19.072 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:59:19.072 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:59:24.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:24.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:24.073 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:59:24.118 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:59:24.119 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:59:24.352 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:59:24.353 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:59:29.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:29.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:29.354 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:59:29.395 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:59:29.395 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:59:29.648 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:59:29.648 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:59:34.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:34.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:34.650 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:59:34.694 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:59:34.695 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:59:34.950 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:59:34.951 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:59:39.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:39.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:39.952 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:59:39.993 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:59:39.993 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:59:40.233 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:59:40.233 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:59:45.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:45.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:45.235 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:59:45.274 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:59:45.274 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:59:45.534 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:59:45.534 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:59:50.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:51.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:51.449 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:59:51.491 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:59:51.492 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:59:51.750 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:59:51.750 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T00:59:56.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:56.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T00:59:56.752 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T00:59:56.792 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T00:59:56.792 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T00:59:57.043 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T00:59:57.044 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:00:02.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:02.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:02.045 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:00:02.086 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:00:02.086 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:00:02.341 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:00:02.341 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:00:07.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:07.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:07.343 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:00:07.383 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:00:07.383 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:00:07.636 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:00:07.636 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:00:12.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:12.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:12.872 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:00:12.911 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:00:12.911 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:00:13.148 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:00:13.149 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:00:18.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:18.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:18.151 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:00:18.191 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:00:18.191 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:00:18.451 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:00:18.451 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:00:23.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:23.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:23.452 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:00:23.494 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:00:23.494 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:00:23.751 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:00:23.751 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:00:28.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:28.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:28.753 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:00:28.794 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:00:28.794 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:00:29.058 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:00:29.058 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:00:34.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:34.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:34.060 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:00:34.101 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:00:34.101 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:00:34.365 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:00:34.366 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:00:39.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:39.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:39.367 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:00:39.413 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:00:39.414 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:00:39.664 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:00:39.664 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:00:44.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:44.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:44.666 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:00:44.707 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:00:44.707 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:00:44.977 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:00:44.978 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:00:49.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:49.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:49.980 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:00:50.022 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:00:50.022 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:00:50.286 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:00:50.407 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:00:55.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:55.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:00:55.409 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:00:55.450 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:00:55.450 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:00:55.713 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:00:55.713 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:01:00.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:00.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:00.715 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:01:00.755 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:01:00.755 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:01:01.015 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:01:01.016 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:01:06.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:06.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:06.017 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:01:06.060 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:01:06.060 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:01:06.325 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:01:06.326 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:01:11.326 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:11.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:11.327 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:01:11.367 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:01:11.367 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:01:11.632 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:01:11.632 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:01:16.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:16.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:16.634 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:01:16.676 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:01:16.676 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:01:16.935 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:01:16.935 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:01:21.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:21.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:21.937 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:01:21.980 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:01:21.980 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:01:22.246 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:01:22.247 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:01:27.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:27.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:27.248 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:01:27.289 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:01:27.289 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:01:27.551 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:01:27.551 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:01:32.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:32.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:32.553 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:01:32.594 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:01:32.594 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:01:32.865 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:01:32.866 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613004976.507018_460.1 target=1_1613003977.863186_285.1 2021-02-11T01:01:37.889 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... FAIL 2021-02-11T01:01:37.890 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-65 2021-02-11T01:01:37.941 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-66 2021-02-11T01:01:38.038 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-67 2021-02-11T01:01:40.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T01:01:40.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T01:01:40.367 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T01:01:40.417 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:01:40.417 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:01:40.624 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T01:01:40.623+0000 7f5a0a13f080 1 bucket sync caught up with source: 2021-02-11T01:01:40.624 INFO:teuthology.orchestra.run.smithi095.stderr: local status: [, , , , , 00000000011.53.5, , , , , ] 2021-02-11T01:01:40.624 INFO:teuthology.orchestra.run.smithi095.stderr: remote markers: [, , , , , 00000000011.53.5, , , , , ] 2021-02-11T01:01:40.624 INFO:teuthology.orchestra.run.smithi095.stderr:2021-02-11T01:01:40.623+0000 7f5a0a13f080 0 bucket checkpoint complete 2021-02-11T01:01:40.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T01:01:40.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T01:01:40.653 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket ufpbfn-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-02-11T01:01:40.704 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:01:40.704 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:01:40.964 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T01:01:40.959+0000 7f08bf8aa080 1 bucket sync caught up with source: 2021-02-11T01:01:40.964 INFO:teuthology.orchestra.run.smithi057.stderr: local status: [, , , , , 00000000011.53.5, , , , , ] 2021-02-11T01:01:40.964 INFO:teuthology.orchestra.run.smithi057.stderr: remote markers: [, , , , , 00000000011.53.5, , , , , ] 2021-02-11T01:01:40.964 INFO:teuthology.orchestra.run.smithi057.stderr:2021-02-11T01:01:40.959+0000 7f08bf8aa080 0 bucket checkpoint complete 2021-02-11T01:01:40.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T01:01:40.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'ufpbfn-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-02-11T01:01:40.994 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket ufpbfn-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-02-11T01:01:41.042 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:01:41.042 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:01:41.250 INFO:teuthology.orchestra.run.smithi095.stderr:could not get bucket info for bucket=ufpbfn-65 2021-02-11T01:01:41.285 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-11T01:01:41.291 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... FAIL 2021-02-11T01:01:41.291 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-68 2021-02-11T01:01:41.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:41.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:41.423 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:01:41.464 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:01:41.464 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:01:41.641 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T01:01:41.673 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005301.331154_487.1","last_update":"2021-02-11T01:01:41.331154Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T01:01:41.673 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T01:01:41.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:41.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:41.674 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:01:41.718 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:01:41.719 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:01:41.954 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005301.331154_487.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:01:41.331154Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:01:41.955 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T01:01:41.955 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T01:01:41.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:41.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:41.955 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:01:41.994 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:01:41.994 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:01:42.237 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005301.331154_487.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:01:41.331154Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:01:42.237 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T01:01:42.237 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T01:01:42.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:42.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:42.238 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:01:42.275 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:01:42.275 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:01:42.526 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:01:42.526 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:01:47.526 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:47.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:47.527 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:01:47.566 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:01:47.566 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:01:47.818 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:01:47.818 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:01:52.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:52.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:52.820 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:01:52.859 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:01:52.859 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:01:53.126 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:01:53.126 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:01:58.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:58.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:01:58.128 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:01:58.168 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:01:58.169 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:01:58.443 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:01:58.443 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:02:03.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:03.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:03.445 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:02:03.486 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:02:03.486 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:02:03.752 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:02:03.752 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:02:08.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:08.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:08.754 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:02:08.800 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:02:08.800 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:02:09.067 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:02:09.068 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:02:14.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:14.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:14.070 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:02:14.111 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:02:14.111 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:02:14.365 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:02:14.366 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:02:19.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:19.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:19.368 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:02:19.408 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:02:19.409 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:02:19.659 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:02:19.659 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:02:24.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:24.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:24.661 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:02:24.704 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:02:24.704 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:02:24.954 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:02:24.954 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:02:29.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:29.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:29.955 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:02:29.995 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:02:29.995 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:02:30.246 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:02:30.247 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:02:35.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:35.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:35.248 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:02:35.281 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:02:35.281 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:02:35.537 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:02:35.537 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:02:40.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:40.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:40.539 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:02:40.579 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:02:40.579 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:02:40.844 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:02:40.844 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:02:45.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:45.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:45.846 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:02:45.886 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:02:45.887 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:02:46.140 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:02:46.141 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:02:51.142 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:51.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:51.143 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:02:51.183 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:02:51.183 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:02:51.434 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:02:51.435 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:02:56.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:56.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:02:56.436 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:02:56.476 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:02:56.476 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:02:56.743 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:02:56.743 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:03:01.744 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:01.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:01.745 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:03:01.786 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:03:01.787 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:03:02.037 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:03:02.037 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:03:07.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:07.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:07.039 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:03:07.084 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:03:07.084 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:03:07.320 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:03:07.320 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:03:12.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:12.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:12.322 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:03:12.364 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:03:12.364 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:03:12.627 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:03:12.627 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:03:17.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:17.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:17.628 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:03:17.671 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:03:17.671 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:03:17.919 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:03:17.920 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:03:22.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:22.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:22.921 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:03:22.963 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:03:22.964 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:03:23.201 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:03:23.202 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:03:28.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:28.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:28.203 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:03:28.245 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:03:28.245 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:03:28.511 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:03:28.511 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:03:33.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:33.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:33.513 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:03:33.555 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:03:33.555 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:03:33.829 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:03:33.830 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:03:38.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:38.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:38.831 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:03:38.873 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:03:38.873 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:03:39.119 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:03:39.120 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:03:44.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:44.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:44.122 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:03:44.160 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:03:44.160 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:03:44.415 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:03:44.416 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:03:49.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:49.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:49.418 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:03:49.460 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:03:49.460 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:03:49.739 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:03:49.739 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:03:54.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:54.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:03:54.741 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:03:54.782 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:03:54.783 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:03:55.061 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:03:57.062 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:04:02.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:02.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:02.063 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:04:02.106 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:04:02.107 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:04:02.383 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:04:02.384 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:04:07.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:07.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:07.386 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:04:07.426 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:04:07.427 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:04:07.688 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:04:07.688 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:04:12.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:12.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:12.690 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:04:12.730 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:04:12.731 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:04:12.997 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:04:12.997 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:04:17.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:17.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:17.998 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:04:18.040 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:04:18.040 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:04:18.304 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:04:18.304 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:04:23.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:23.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:23.306 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:04:23.348 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:04:23.348 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:04:23.614 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:04:23.615 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:04:28.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:28.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:28.616 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:04:28.657 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:04:28.658 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:04:28.923 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:04:28.924 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:04:33.924 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:33.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:33.925 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:04:33.972 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:04:33.972 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:04:34.225 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:04:34.225 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:04:39.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:39.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:39.227 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:04:39.268 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:04:39.269 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:04:39.526 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:04:39.526 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:04:44.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:44.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:44.528 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:04:44.567 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:04:44.567 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:04:44.838 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:04:44.839 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:04:49.840 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:49.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:49.840 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:04:49.883 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:04:49.883 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:04:50.149 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:04:50.150 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:04:55.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:55.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:04:55.151 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:04:55.190 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:04:55.190 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:04:55.441 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:04:55.441 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:05:00.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:00.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:00.442 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:05:00.484 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:05:00.484 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:05:00.741 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:05:00.741 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:05:05.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:05.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:05.742 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:05:05.785 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:05:05.785 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:05:06.050 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:05:06.050 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:05:11.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:11.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:11.051 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:05:11.092 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:05:11.092 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:05:11.337 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:05:11.338 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:05:16.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:16.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:16.340 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:05:16.379 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:05:16.379 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:05:16.638 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:05:16.638 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:05:21.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:21.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:21.639 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:05:21.682 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:05:21.682 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:05:21.948 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:05:21.948 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:05:26.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:26.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:26.949 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:05:26.991 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:05:26.991 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:05:27.253 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:05:27.254 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:05:32.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:32.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:32.255 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:05:32.294 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:05:32.295 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:05:32.563 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:05:32.563 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:05:37.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:37.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:37.564 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:05:37.606 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:05:37.606 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:05:37.872 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:05:37.872 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:05:42.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:42.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:42.874 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:05:42.915 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:05:42.915 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:05:43.176 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:05:43.176 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:05:48.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:48.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:48.178 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:05:48.219 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:05:48.220 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:05:48.494 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:05:48.494 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:05:53.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:53.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:53.496 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:05:53.539 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:05:53.539 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:05:53.810 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:05:53.811 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:05:58.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:58.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:05:58.812 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:05:58.855 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:05:58.855 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:05:59.118 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:05:59.118 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:06:04.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:04.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:04.120 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:06:04.154 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:06:04.154 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:06:04.412 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:06:04.412 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:06:09.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:09.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:09.414 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:06:09.457 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:06:09.457 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:06:09.731 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:06:09.731 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:06:14.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:14.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:14.733 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:06:14.774 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:06:14.774 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:06:15.052 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:06:15.052 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:06:20.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:20.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:20.054 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:06:20.094 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:06:20.094 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:06:20.334 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:06:20.334 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:06:25.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:25.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:25.336 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:06:25.375 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:06:25.375 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:06:25.633 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:06:25.633 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:06:30.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:30.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:30.636 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:06:30.678 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:06:30.679 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:06:30.933 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:06:30.934 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:06:35.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:35.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:35.935 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:06:35.977 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:06:35.977 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:06:36.232 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:06:36.232 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:06:41.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:41.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:41.234 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:06:41.275 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:06:41.275 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:06:41.556 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:06:41.557 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:06:46.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:46.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:46.558 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:06:46.601 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:06:46.602 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:06:46.865 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:06:46.866 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:06:51.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:51.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:51.868 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:06:51.907 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:06:51.907 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:06:52.168 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:06:52.168 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:06:57.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:57.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:06:57.170 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:06:57.204 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:06:57.204 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:06:57.456 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:06:57.456 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005301.331154_487.1 target=1_1613003977.863186_285.1 2021-02-11T01:07:02.480 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... FAIL 2021-02-11T01:07:02.480 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-69 2021-02-11T01:07:02.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:02.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:02.723 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:07:02.766 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:07:02.766 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:07:02.954 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T01:07:02.987 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005622.526776_500.1","last_update":"2021-02-11T01:07:02.526776Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T01:07:02.988 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T01:07:02.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:02.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:02.988 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:07:03.025 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:07:03.026 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:07:03.253 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005622.526776_500.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:07:02.526776Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:07:03.253 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T01:07:03.253 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T01:07:03.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:03.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:03.254 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:07:03.295 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:07:03.296 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:07:03.548 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005622.526776_500.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:07:02.526776Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:07:03.549 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T01:07:03.549 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T01:07:03.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:03.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:03.549 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:07:03.590 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:07:03.590 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:07:03.854 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:07:03.855 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:07:08.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:08.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:08.857 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:07:08.897 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:07:08.897 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:07:09.145 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:07:09.145 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:07:14.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:14.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:14.146 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:07:14.188 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:07:14.188 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:07:14.447 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:07:14.448 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:07:19.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:19.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:19.449 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:07:19.491 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:07:19.492 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:07:19.770 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:07:19.771 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:07:24.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:24.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:24.772 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:07:24.813 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:07:24.813 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:07:25.076 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:07:25.077 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:07:30.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:30.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:30.078 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:07:30.119 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:07:30.119 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:07:30.370 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:07:30.371 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:07:35.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:35.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:35.372 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:07:35.412 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:07:35.413 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:07:35.691 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:07:37.217 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:07:42.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:42.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:42.218 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:07:42.259 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:07:42.260 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:07:42.528 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:07:42.529 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:07:47.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:47.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:47.531 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:07:47.572 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:07:47.573 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:07:47.822 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:07:47.823 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:07:52.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:52.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:52.824 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:07:52.867 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:07:52.867 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:07:53.128 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:07:53.128 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:07:58.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:58.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:07:58.130 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:07:58.172 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:07:58.172 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:07:58.425 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:07:58.425 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:08:03.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:08:03.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:08:03.427 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:08:03.469 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:08:03.469 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:08:03.722 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:08:03.723 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:08:08.723 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:08:08.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:08:08.724 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:08:08.766 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:08:08.766 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:08:09.025 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:08:09.026 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:08:14.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:08:14.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:08:14.028 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:08:14.069 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:08:14.069 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:08:14.329 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:08:14.330 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:08:19.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:08:24.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:08:24.013 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:08:24.055 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:08:24.055 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:08:24.305 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:08:24.306 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:08:29.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:08:29.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:08:29.308 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:08:29.350 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:08:29.350 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:08:29.607 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:08:29.608 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:08:34.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:08:34.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:08:34.609 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:08:34.652 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:08:34.652 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:08:34.929 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:08:43.132 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:08:48.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:08:48.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:08:48.134 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:08:48.176 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:08:48.176 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:08:48.434 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:08:48.434 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:08:53.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:08:55.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:08:55.341 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:08:55.383 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:08:55.383 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:08:55.654 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:08:55.655 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:09:00.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:00.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:00.656 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:09:00.697 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:09:00.697 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:09:00.964 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:09:00.965 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:09:05.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:08.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:08.447 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:09:08.491 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:09:08.491 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:09:08.758 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:09:08.759 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:09:13.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:13.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:13.761 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:09:13.803 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:09:13.803 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:09:14.067 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:09:14.068 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:09:19.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:19.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:19.069 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:09:19.115 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:09:19.115 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:09:19.380 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:09:19.381 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:09:24.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:24.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:24.382 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:09:24.424 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:09:24.424 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:09:24.698 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:09:24.698 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:09:29.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:32.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:32.085 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:09:32.129 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:09:32.129 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:09:32.364 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:09:32.365 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:09:37.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:37.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:37.367 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:09:37.408 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:09:37.409 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:09:37.666 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:09:37.666 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:09:42.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:42.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:42.668 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:09:42.709 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:09:42.709 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:09:42.978 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:09:42.978 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:09:47.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:47.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:47.980 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:09:48.021 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:09:48.021 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:09:48.274 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:09:48.275 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:09:53.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:53.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:53.276 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:09:53.317 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:09:53.317 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:09:53.586 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:09:53.587 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:09:58.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:58.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:09:58.588 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:09:58.627 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:09:58.627 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:09:58.897 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:09:58.898 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:10:03.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:03.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:03.899 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:10:03.939 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:10:03.939 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:10:04.178 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:10:04.178 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:10:09.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:10.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:10.382 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:10:10.422 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:10:10.423 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:10:10.680 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:10:10.680 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:10:15.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:15.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:15.682 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:10:15.724 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:10:15.724 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:10:15.981 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:10:15.982 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:10:20.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:20.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:20.983 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:10:21.024 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:10:21.025 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:10:21.272 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:10:21.273 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:10:26.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:26.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:26.275 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:10:26.314 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:10:26.315 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:10:26.569 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:10:26.569 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:10:31.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:31.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:31.571 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:10:31.613 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:10:31.614 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:10:31.873 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:10:31.874 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:10:36.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:36.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:36.875 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:10:36.917 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:10:36.917 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:10:37.169 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:10:37.170 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:10:42.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:45.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:45.689 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:10:45.731 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:10:45.731 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:10:46.020 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:10:46.021 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:10:51.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:51.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:51.022 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:10:51.068 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:10:51.068 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:10:51.296 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:10:51.297 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:10:56.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:56.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:10:56.298 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:10:56.338 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:10:56.338 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:10:56.598 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:10:56.598 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:11:01.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:09.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:09.085 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:11:09.128 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:11:09.128 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:11:09.404 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:11:09.404 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:11:14.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:14.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:14.406 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:11:14.446 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:11:14.446 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:11:14.706 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:11:14.707 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:11:19.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:19.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:19.709 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:11:19.750 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:11:19.750 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:11:20.009 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:11:20.010 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:11:25.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:25.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:25.011 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:11:25.050 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:11:25.050 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:11:25.309 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:11:25.309 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:11:30.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:30.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:30.311 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:11:30.350 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:11:30.350 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:11:30.604 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:11:30.604 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:11:35.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:35.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:35.605 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:11:35.647 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:11:35.647 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:11:35.909 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:11:35.909 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:11:40.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:40.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:40.911 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:11:40.952 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:11:40.952 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:11:41.200 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:11:41.200 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:11:46.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:46.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:46.202 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:11:46.242 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:11:46.242 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:11:46.518 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:11:46.519 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:11:51.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:51.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:51.521 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:11:51.563 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:11:51.563 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:11:51.828 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:11:51.828 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:11:56.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:56.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:11:56.830 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:11:56.870 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:11:56.870 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:11:57.116 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:11:57.117 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:12:02.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:02.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:02.118 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:12:02.158 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:12:02.158 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:12:02.412 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:12:02.413 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:12:07.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:07.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:07.414 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:12:07.456 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:12:07.457 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:12:07.717 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:12:07.718 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:12:12.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:14.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:14.082 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:12:14.122 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:12:14.122 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:12:14.380 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:12:14.381 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:12:19.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:19.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:19.382 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:12:19.422 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:12:19.423 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:12:19.701 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:12:19.702 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:12:24.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:24.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:24.703 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:12:24.742 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:12:24.742 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:12:24.999 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:12:24.999 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:12:30.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:30.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:30.001 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:12:30.043 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:12:30.043 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:12:30.313 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:12:30.314 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:12:35.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:35.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:35.315 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:12:35.354 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:12:35.354 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:12:35.615 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:12:35.615 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:12:40.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:40.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:40.617 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:12:40.658 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:12:40.659 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:12:40.931 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:12:40.931 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:12:45.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:49.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:49.833 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:12:49.873 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:12:49.874 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:12:50.155 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:12:50.155 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:12:55.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:55.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:12:55.156 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:12:55.195 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:12:55.196 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:12:55.445 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:12:55.446 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005622.526776_500.1 target=1_1613003977.863186_285.1 2021-02-11T01:13:00.467 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... FAIL 2021-02-11T01:13:00.518 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... ok 2021-02-11T01:13:00.519 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=ufpbfn-70 2021-02-11T01:13:00.561 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=ufpbfn-71 2021-02-11T01:13:00.643 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=ufpbfn-72 2021-02-11T01:13:00.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:00.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:00.851 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:13:00.887 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:13:00.887 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:13:01.061 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T01:13:01.098 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T01:13:01.098 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T01:13:01.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:01.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:01.099 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:13:01.137 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:13:01.137 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:13:01.368 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:13:01.369 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T01:13:01.369 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T01:13:01.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:01.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:01.369 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:13:01.410 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:13:01.410 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:13:01.673 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:13:01.673 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T01:13:01.673 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T01:13:01.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:01.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:01.674 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:13:01.714 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:13:01.714 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:13:01.995 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:13:01.995 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:13:06.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:06.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:06.996 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:13:07.037 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:13:07.037 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:13:07.291 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:13:07.292 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:13:12.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:12.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:12.293 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:13:12.330 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:13:12.330 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:13:12.594 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:13:12.594 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:13:17.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:17.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:17.596 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:13:17.639 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:13:17.639 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:13:17.892 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:13:17.892 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:13:22.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:22.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:22.894 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:13:22.933 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:13:22.933 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:13:23.226 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:13:23.227 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:13:28.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:28.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:28.228 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:13:28.269 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:13:28.269 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:13:28.525 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:13:28.525 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:13:33.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:33.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:33.526 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:13:33.566 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:13:33.566 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:13:33.833 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:13:33.833 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:13:38.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:38.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:38.835 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:13:38.875 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:13:38.875 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:13:39.145 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:13:39.145 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:13:44.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:44.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:44.147 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:13:44.183 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:13:44.183 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:13:44.426 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:13:44.427 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:13:49.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:49.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:49.428 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:13:49.466 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:13:49.466 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:13:49.724 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:13:49.725 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:13:54.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:54.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:13:54.727 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:13:54.768 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:13:54.768 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:13:55.044 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:13:55.045 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:14:00.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:02.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:02.401 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:14:02.441 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:14:02.441 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:14:02.720 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:14:02.721 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:14:07.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:07.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:07.722 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:14:07.762 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:14:07.763 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:14:08.026 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:14:08.027 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:14:13.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:13.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:13.028 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:14:13.068 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:14:13.068 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:14:13.349 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:14:13.349 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:14:18.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:18.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:18.351 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:14:18.392 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:14:18.392 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:14:18.660 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:14:18.660 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:14:23.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:23.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:23.662 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:14:23.704 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:14:23.704 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:14:23.955 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:14:23.955 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:14:28.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:28.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:28.956 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:14:28.997 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:14:28.997 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:14:29.261 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:14:29.261 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:14:34.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:36.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:36.277 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:14:36.320 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:14:36.320 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:14:36.587 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:14:36.587 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:14:41.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:41.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:41.589 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:14:41.629 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:14:41.629 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:14:41.895 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:14:41.896 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:14:46.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:46.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:46.897 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:14:46.938 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:14:46.938 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:14:47.178 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:14:47.178 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:14:52.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:52.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:52.180 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:14:52.221 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:14:52.222 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:14:52.504 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:14:52.505 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:14:57.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:57.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:14:57.507 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:14:57.549 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:14:57.549 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:14:57.813 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:14:57.813 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:15:02.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:02.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:02.815 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:15:02.854 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:15:02.854 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:15:03.119 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:15:03.120 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:15:08.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:14.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:14.862 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:15:14.902 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:15:14.902 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:15:15.168 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:15:15.168 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:15:20.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:20.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:20.170 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:15:20.209 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:15:20.209 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:15:20.473 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:15:20.473 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:15:25.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:25.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:25.475 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:15:25.515 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:15:25.515 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:15:25.773 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:15:25.773 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:15:30.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:30.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:30.775 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:15:30.815 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:15:30.815 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:15:31.078 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:15:31.079 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:15:36.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:36.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:36.080 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:15:36.131 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:15:36.131 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:15:36.362 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:15:36.362 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:15:41.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:41.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:41.364 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:15:41.407 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:15:41.407 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:15:41.670 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:15:41.670 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:15:46.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:46.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:46.672 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:15:46.714 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:15:46.715 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:15:46.968 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:15:46.968 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:15:51.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:51.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:51.970 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:15:52.010 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:15:52.010 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:15:52.261 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:15:52.261 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:15:57.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:57.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:15:57.263 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:15:57.304 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:15:57.304 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:15:57.589 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:15:57.590 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:16:02.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:02.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:02.591 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:16:02.630 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:16:02.630 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:16:02.902 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:16:02.902 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:16:07.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:07.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:07.904 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:16:07.946 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:16:07.947 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:16:08.197 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:16:08.197 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:16:13.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:13.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:13.199 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:16:13.239 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:16:13.239 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:16:13.506 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:16:13.507 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:16:18.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:20.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:20.026 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:16:20.067 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:16:20.067 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:16:20.312 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:16:20.313 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:16:25.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:25.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:25.315 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:16:25.354 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:16:25.354 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:16:25.619 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:16:25.619 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:16:30.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:30.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:30.620 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:16:30.661 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:16:30.661 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:16:30.917 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:16:30.918 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:16:35.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:35.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:35.919 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:16:35.961 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:16:35.961 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:16:36.250 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:16:36.250 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:16:41.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:41.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:41.251 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:16:41.293 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:16:41.293 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:16:41.574 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:16:41.574 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:16:46.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:46.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:46.576 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:16:46.617 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:16:46.617 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:16:46.875 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:16:46.875 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:16:51.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:52.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:52.199 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:16:52.239 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:16:52.239 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:16:52.491 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:16:52.491 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:16:57.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:57.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:16:57.494 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:16:57.535 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:16:57.536 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:16:57.820 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:16:57.820 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:17:02.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:02.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:02.821 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:17:02.863 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:17:02.863 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:17:03.123 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:17:03.123 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:17:08.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:08.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:08.125 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:17:08.167 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:17:08.167 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:17:08.418 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:17:08.418 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:17:13.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:13.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:13.420 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:17:13.462 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:17:13.462 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:17:13.730 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:17:13.730 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:17:18.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:18.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:18.732 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:17:18.774 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:17:18.775 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:17:19.046 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:17:19.047 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:17:24.048 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:24.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:24.048 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:17:24.091 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:17:24.091 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:17:24.330 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:17:24.331 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:17:29.332 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:33.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:33.543 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:17:33.587 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:17:33.587 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:17:33.851 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:17:33.851 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:17:38.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:38.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:38.852 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:17:38.895 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:17:38.895 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:17:39.145 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:17:39.145 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:17:44.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:44.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:44.147 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:17:44.182 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:17:44.182 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:17:44.427 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:17:44.428 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:17:49.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:49.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:49.429 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:17:49.471 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:17:49.471 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:17:49.741 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:17:49.742 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:17:54.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:54.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:17:54.743 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:17:54.783 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:17:54.783 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:17:55.063 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:17:55.064 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:18:00.064 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:00.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:00.065 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:18:00.103 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:18:00.104 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:18:00.344 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:18:00.344 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:18:05.345 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:05.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:05.346 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:18:05.386 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:18:05.386 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:18:05.646 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:18:05.647 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:18:10.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:10.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:10.648 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:18:10.689 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:18:10.689 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:18:10.964 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:18:10.965 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:18:15.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:15.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:15.967 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:18:16.006 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:18:16.006 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:18:16.267 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:18:16.268 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:18:21.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:21.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:21.269 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:18:21.309 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:18:21.310 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:18:21.578 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:18:21.579 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:18:26.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:26.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:26.580 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:18:26.621 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:18:26.621 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:18:26.897 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:18:26.897 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:18:31.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:31.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:31.898 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:18:31.942 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:18:31.942 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:18:32.200 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:18:32.200 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:18:37.224 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_backing ... FAIL 2021-02-11T01:18:37.225 INFO:tasks.rgw_multisite_tests: 2021-02-11T01:18:37.225 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T01:18:37.226 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_sync_disable 2021-02-11T01:18:37.226 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T01:18:37.226 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T01:18:37.226 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T01:18:37.226 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T01:18:37.226 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 1106, in test_bucket_sync_disable 2021-02-11T01:18:37.226 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T01:18:37.226 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T01:18:37.226 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T01:18:37.226 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T01:18:37.226 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T01:18:37.226 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T01:18:37.226 INFO:tasks.rgw_multisite_tests: 2021-02-11T01:18:37.227 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T01:18:37.227 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable 2021-02-11T01:18:37.227 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T01:18:37.227 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T01:18:37.227 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T01:18:37.227 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T01:18:37.227 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 1129, in test_bucket_sync_enable_right_after_disable 2021-02-11T01:18:37.227 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T01:18:37.227 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T01:18:37.227 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T01:18:37.227 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T01:18:37.227 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T01:18:37.227 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T01:18:37.227 INFO:tasks.rgw_multisite_tests: 2021-02-11T01:18:37.228 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T01:18:37.228 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_sync_disable_enable 2021-02-11T01:18:37.228 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T01:18:37.228 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T01:18:37.228 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T01:18:37.228 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T01:18:37.228 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 1163, in test_bucket_sync_disable_enable 2021-02-11T01:18:37.228 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T01:18:37.228 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T01:18:37.228 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T01:18:37.228 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T01:18:37.229 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T01:18:37.229 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T01:18:37.229 INFO:tasks.rgw_multisite_tests: 2021-02-11T01:18:37.229 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T01:18:37.229 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_multipart_object_sync 2021-02-11T01:18:37.229 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T01:18:37.229 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T01:18:37.229 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T01:18:37.229 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T01:18:37.229 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 1207, in test_multipart_object_sync 2021-02-11T01:18:37.229 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2021-02-11T01:18:37.229 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 387, in zonegroup_bucket_checkpoint 2021-02-11T01:18:37.230 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2021-02-11T01:18:37.230 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 379, in zone_bucket_checkpoint 2021-02-11T01:18:37.230 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2021-02-11T01:18:37.230 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multisite.py", line 202, in admin 2021-02-11T01:18:37.230 INFO:tasks.rgw_multisite_tests: assert r == 0 2021-02-11T01:18:37.230 INFO:tasks.rgw_multisite_tests:AssertionError 2021-02-11T01:18:37.230 INFO:tasks.rgw_multisite_tests: 2021-02-11T01:18:37.230 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T01:18:37.230 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_encrypted_object_sync 2021-02-11T01:18:37.230 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T01:18:37.230 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T01:18:37.230 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T01:18:37.231 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T01:18:37.231 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 1243, in test_encrypted_object_sync 2021-02-11T01:18:37.231 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T01:18:37.231 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T01:18:37.231 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T01:18:37.231 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T01:18:37.231 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T01:18:37.231 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T01:18:37.231 INFO:tasks.rgw_multisite_tests: 2021-02-11T01:18:37.231 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T01:18:37.231 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_index_log_trim 2021-02-11T01:18:37.231 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T01:18:37.232 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T01:18:37.232 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T01:18:37.232 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T01:18:37.232 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 1273, in test_bucket_index_log_trim 2021-02-11T01:18:37.232 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket() 2021-02-11T01:18:37.232 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 1268, in make_test_bucket 2021-02-11T01:18:37.232 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T01:18:37.232 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T01:18:37.232 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T01:18:37.232 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T01:18:37.232 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T01:18:37.232 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T01:18:37.232 INFO:tasks.rgw_multisite_tests: 2021-02-11T01:18:37.233 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T01:18:37.233 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_datalog_backing 2021-02-11T01:18:37.233 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T01:18:37.233 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T01:18:37.233 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T01:18:37.233 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T01:18:37.233 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 1363, in test_datalog_backing 2021-02-11T01:18:37.233 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T01:18:37.233 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T01:18:37.233 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T01:18:37.233 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T01:18:37.233 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T01:18:37.234 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T01:18:37.234 INFO:tasks.rgw_multisite_tests: 2021-02-11T01:18:37.234 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T01:18:37.234 INFO:tasks.rgw_multisite_tests:Ran 29 tests in 2746.469s 2021-02-11T01:18:37.234 INFO:tasks.rgw_multisite_tests: 2021-02-11T01:18:37.234 INFO:tasks.rgw_multisite_tests:FAILED (failures=7) 2021-02-11T01:18:37.241 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2021-02-11T01:18:37.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:37.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:37.242 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:18:37.282 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:18:37.282 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:18:37.500 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T01:18:37.543 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T01:18:39.156 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T01:18:39.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:39.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:39.156 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:18:39.194 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:18:39.194 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:18:39.446 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:18:39.447 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T01:18:39.447 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T01:18:39.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:39.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:39.447 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:18:39.487 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:18:39.488 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:18:39.741 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:18:39.741 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T01:18:39.741 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T01:18:39.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:39.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:39.742 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:18:39.783 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:18:39.783 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:18:40.042 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:18:40.042 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:18:45.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:45.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:45.044 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:18:45.085 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:18:45.085 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:18:45.355 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:18:45.355 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:18:50.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:50.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:50.357 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:18:50.398 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:18:50.398 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:18:50.690 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:18:50.690 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:18:55.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:55.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:18:55.692 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:18:55.732 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:18:55.732 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:18:56.006 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:18:56.007 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:19:01.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:01.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:01.008 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:19:01.050 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:19:01.051 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:19:01.316 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:19:01.316 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:19:06.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:06.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:06.318 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:19:06.358 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:19:06.358 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:19:06.645 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:19:06.645 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:19:11.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:16.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:16.988 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:19:17.029 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:19:17.029 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:19:17.289 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:19:17.289 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:19:22.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:22.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:22.291 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:19:22.332 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:19:22.332 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:19:22.608 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:19:22.608 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:19:27.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:27.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:27.609 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:19:27.650 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:19:27.650 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:19:27.924 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:19:27.925 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:19:32.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:32.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:32.926 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:19:32.966 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:19:32.966 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:19:33.229 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:19:33.230 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:19:38.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:38.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:38.231 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:19:38.272 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:19:38.272 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:19:38.537 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:19:38.537 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:19:43.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:43.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:43.539 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:19:43.580 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:19:43.580 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:19:43.861 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:19:43.861 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:19:48.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:48.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:48.862 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:19:48.904 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:19:48.904 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:19:49.180 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:19:49.180 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:19:54.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:54.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:54.182 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:19:54.223 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:19:54.223 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:19:54.496 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:19:54.496 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:19:59.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:59.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:19:59.498 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:19:59.539 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:19:59.540 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:19:59.805 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:19:59.806 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:20:04.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:04.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:04.807 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:20:04.847 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:20:04.847 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:20:05.107 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:20:05.108 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:20:10.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:10.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:10.109 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:20:10.152 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:20:10.152 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:20:10.413 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:20:10.413 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:20:15.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:15.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:15.415 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:20:15.454 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:20:15.455 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:20:15.727 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:20:15.728 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:20:20.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:26.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:26.968 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:20:27.010 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:20:27.010 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:20:27.280 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:20:27.280 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:20:32.281 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:32.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:32.282 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:20:32.322 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:20:32.322 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:20:32.591 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:20:32.592 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:20:37.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:37.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:37.593 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:20:37.635 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:20:37.635 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:20:37.913 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:20:37.914 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:20:42.914 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:42.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:42.915 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:20:42.955 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:20:42.955 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:20:43.207 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:20:43.208 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:20:48.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:48.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:48.209 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:20:48.251 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:20:48.251 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:20:48.509 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:20:48.510 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:20:53.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:53.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:20:53.512 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:20:53.554 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:20:53.555 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:20:53.822 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:20:53.823 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:20:58.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:01.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:01.218 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:21:01.258 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:21:01.258 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:21:01.523 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:21:01.523 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:21:06.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:06.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:06.525 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:21:06.567 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:21:06.567 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:21:06.841 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:21:06.842 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:21:11.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:11.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:11.844 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:21:11.884 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:21:11.885 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:21:12.157 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:21:12.158 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:21:17.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:17.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:17.160 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:21:17.201 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:21:17.202 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:21:17.464 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:21:17.465 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:21:22.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:22.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:22.466 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:21:22.507 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:21:22.507 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:21:22.781 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:21:22.782 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:21:27.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:27.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:27.784 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:21:27.825 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:21:27.825 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:21:28.087 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:21:28.087 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:21:33.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:38.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:38.681 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:21:38.721 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:21:38.721 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:21:38.994 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:21:38.995 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:21:43.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:43.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:43.997 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:21:44.037 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:21:44.038 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:21:44.295 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:21:44.295 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:21:49.297 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:49.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:49.297 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:21:49.337 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:21:49.338 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:21:49.614 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:21:49.615 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:21:54.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:54.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:54.616 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:21:54.657 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:21:54.657 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:21:54.923 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:21:54.924 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:21:59.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:59.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:21:59.926 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:21:59.966 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:21:59.966 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:22:00.193 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:22:00.193 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:22:05.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:05.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:05.194 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:22:05.236 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:22:05.236 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:22:05.521 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:22:05.521 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:22:10.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:13.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:13.565 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:22:13.606 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:22:13.606 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:22:13.889 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:22:13.890 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:22:18.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:18.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:18.892 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:22:18.934 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:22:18.934 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:22:19.198 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:22:19.199 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:22:24.199 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:24.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:24.200 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:22:24.241 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:22:24.242 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:22:24.514 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:22:24.514 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:22:29.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:29.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:29.515 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:22:29.555 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:22:29.555 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:22:29.831 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:22:29.832 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:22:34.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:34.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:34.833 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:22:34.874 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:22:34.875 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:22:35.130 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:22:35.130 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:22:40.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:40.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:40.132 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:22:40.173 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:22:40.174 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:22:40.444 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:22:40.444 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:22:45.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:45.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:45.446 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:22:45.488 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:22:45.488 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:22:45.765 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:22:45.765 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:22:50.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:50.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:50.767 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:22:50.808 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:22:50.808 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:22:51.070 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:22:51.071 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:22:56.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:56.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:22:56.073 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:22:56.115 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:22:56.115 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:22:56.404 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:22:56.404 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:23:01.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:01.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:01.406 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:23:01.447 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:23:01.447 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:23:01.703 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:23:01.703 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:23:06.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:06.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:06.705 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:23:06.746 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:23:06.746 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:23:07.023 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:23:07.023 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:23:12.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:12.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:12.025 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:23:12.067 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:23:12.067 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:23:12.324 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:23:12.325 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:23:17.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:17.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:17.326 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:23:17.368 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:23:17.368 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:23:17.648 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:23:17.648 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:23:22.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:22.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:22.650 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:23:22.693 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:23:22.693 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:23:22.957 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:23:22.958 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:23:27.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:27.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:27.960 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:23:28.000 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:23:28.001 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:23:28.277 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:23:28.278 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:23:33.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:33.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:33.279 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:23:33.320 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:23:33.321 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:23:33.590 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:23:33.590 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:23:38.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:38.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:38.591 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:23:38.634 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:23:38.634 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:23:38.892 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:23:38.892 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:23:43.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:43.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:43.894 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:23:43.936 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:23:43.936 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:23:44.210 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:23:44.210 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:23:49.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:49.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:49.211 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:23:49.251 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:23:49.251 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:23:49.521 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:23:49.521 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:23:54.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:54.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:54.523 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:23:54.562 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:23:54.563 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:23:54.833 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:23:54.834 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:23:59.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:59.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:23:59.835 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:23:59.876 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:23:59.876 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:24:00.135 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:24:00.135 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:24:05.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:05.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:05.136 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:24:05.178 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:24:05.178 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:24:05.429 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:24:05.429 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:24:10.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:10.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:10.431 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:24:10.471 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:24:10.471 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:24:10.731 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:24:10.731 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:24:15.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:15.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:15.732 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:24:15.773 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:24:15.773 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:24:16.046 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:24:16.047 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:24:21.074 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... FAIL 2021-02-11T01:24:21.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:21.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:21.075 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:24:21.122 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:24:21.122 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:24:21.329 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T01:24:21.370 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T01:24:21.370 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T01:24:21.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:21.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:21.371 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:24:21.410 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:24:21.410 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:24:21.678 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:24:21.678 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T01:24:21.678 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T01:24:21.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:21.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:21.679 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:24:21.721 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:24:21.721 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:24:21.986 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:24:21.987 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T01:24:21.987 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T01:24:21.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:21.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:21.987 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:24:22.028 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:24:22.028 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:24:22.291 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:24:22.291 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:24:27.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:27.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:27.293 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:24:27.332 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:24:27.332 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:24:27.601 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:24:27.601 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:24:32.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:32.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:32.603 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:24:32.643 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:24:32.643 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:24:32.915 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:24:32.916 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:24:37.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:37.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:37.917 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:24:37.957 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:24:37.958 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:24:38.227 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:24:38.228 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:24:43.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:43.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:43.229 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:24:43.272 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:24:43.272 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:24:43.535 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:24:43.535 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:24:48.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:48.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:48.537 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:24:48.577 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:24:48.577 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:24:48.849 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:24:48.849 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:24:53.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:53.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:53.850 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:24:53.892 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:24:53.893 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:24:54.160 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:24:54.161 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:24:59.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:59.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:24:59.163 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:24:59.203 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:24:59.203 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:24:59.463 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:24:59.464 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:25:04.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:04.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:04.466 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:25:04.506 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:25:04.507 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:25:04.784 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:25:04.784 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:25:09.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:09.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:09.785 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:25:09.827 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:25:09.827 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:25:10.089 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:25:10.090 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:25:15.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:15.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:15.092 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:25:15.127 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:25:15.127 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:25:15.368 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:25:15.369 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:25:20.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:20.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:20.370 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:25:20.412 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:25:20.412 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:25:20.685 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:25:20.685 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:25:25.686 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:25.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:25.686 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:25:25.726 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:25:25.726 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:25:26.005 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:25:26.006 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:25:31.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:31.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:31.007 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:25:31.048 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:25:31.048 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:25:31.298 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:25:31.299 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:25:36.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:36.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:36.300 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:25:36.342 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:25:36.342 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:25:36.598 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:25:36.598 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:25:41.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:41.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:41.600 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:25:41.641 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:25:41.641 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:25:41.913 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:25:41.913 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:25:46.914 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:46.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:46.914 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:25:46.956 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:25:46.956 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:25:47.208 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:25:47.209 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:25:52.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:52.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:52.211 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:25:52.252 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:25:52.252 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:25:52.516 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:25:52.517 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:25:57.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:57.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:25:57.518 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:25:57.561 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:25:57.561 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:25:57.856 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:25:57.856 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:26:02.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:02.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:02.857 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:26:02.899 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:26:02.899 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:26:03.159 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:26:03.159 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:26:08.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:08.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:08.161 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:26:08.202 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:26:08.202 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:26:08.471 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:26:08.472 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:26:13.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:13.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:13.473 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:26:13.514 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:26:13.514 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:26:13.790 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:26:13.791 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:26:18.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:20.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:20.440 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:26:20.482 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:26:20.482 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:26:20.752 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:26:20.752 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:26:25.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:25.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:25.753 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:26:25.795 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:26:25.795 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:26:26.071 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:26:26.072 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:26:31.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:31.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:31.073 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:26:31.114 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:26:31.114 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:26:31.370 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:26:31.371 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:26:36.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:36.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:36.372 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:26:36.415 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:26:36.415 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:26:36.675 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:26:36.675 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:26:41.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:41.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:41.677 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:26:41.718 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:26:41.718 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:26:41.977 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:26:41.978 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:26:46.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:46.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:46.979 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:26:47.018 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:26:47.018 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:26:47.288 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:26:47.288 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:26:52.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:52.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:52.533 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:26:52.576 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:26:52.576 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:26:52.859 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:26:52.859 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:26:57.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:57.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:26:57.861 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:26:57.903 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:26:57.903 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:26:58.182 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:26:58.183 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:27:03.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:03.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:03.184 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:27:03.222 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:27:03.222 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:27:03.493 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:27:03.493 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:27:08.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:08.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:08.494 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:27:08.534 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:27:08.534 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:27:08.804 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:27:08.805 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:27:13.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:13.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:13.806 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:27:13.846 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:27:13.846 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:27:14.107 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:27:14.108 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:27:19.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:19.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:19.109 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:27:19.150 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:27:19.151 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:27:19.410 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:27:19.410 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:27:24.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:24.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:24.411 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:27:24.451 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:27:24.451 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:27:24.728 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:27:24.728 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:27:29.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:29.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:29.729 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:27:29.772 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:27:29.772 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:27:30.040 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:27:30.040 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:27:35.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:35.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:35.042 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:27:35.085 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:27:35.086 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:27:35.337 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:27:35.337 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:27:40.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:40.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:40.338 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:27:40.377 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:27:40.377 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:27:40.653 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:27:40.654 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:27:45.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:45.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:45.656 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:27:45.699 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:27:45.699 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:27:45.979 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:27:45.980 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:27:50.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:50.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:50.981 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:27:51.023 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:27:51.024 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:27:51.267 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:27:51.268 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:27:56.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:56.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:27:56.270 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:27:56.310 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:27:56.310 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:27:56.573 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:27:56.573 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:28:01.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:02.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:02.937 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:28:02.977 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:28:02.977 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:28:03.257 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:28:03.257 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:28:08.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:08.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:08.259 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:28:08.297 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:28:08.297 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:28:08.564 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:28:08.564 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:28:13.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:13.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:13.565 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:28:13.608 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:28:13.608 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:28:13.877 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:28:13.877 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:28:18.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:18.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:18.879 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:28:18.919 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:28:18.919 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:28:19.197 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:28:19.197 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:28:24.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:24.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:24.199 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:28:24.236 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:28:24.236 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:28:24.518 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:28:24.518 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:28:29.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:29.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:29.520 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:28:29.560 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:28:29.560 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:28:29.825 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:28:29.825 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:28:34.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:34.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:34.826 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:28:34.868 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:28:34.868 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:28:35.131 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:28:35.132 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:28:40.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:40.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:40.133 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:28:40.175 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:28:40.175 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:28:40.415 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:28:40.415 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:28:45.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:45.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:45.416 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:28:45.459 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:28:45.459 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:28:45.737 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:28:45.737 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:28:50.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:50.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:50.739 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:28:50.781 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:28:50.781 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:28:51.044 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:28:51.044 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:28:56.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:56.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:28:56.045 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:28:56.092 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:28:56.092 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:28:56.353 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:28:56.354 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:29:01.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:01.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:01.355 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:29:01.395 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:29:01.395 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:29:01.675 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:29:01.675 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:29:06.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:06.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:06.676 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:29:06.717 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:29:06.717 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:29:06.981 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:29:06.981 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:29:11.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:11.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:11.983 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:29:12.025 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:29:12.025 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:29:12.289 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:29:12.290 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:29:17.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:17.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:17.291 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:29:17.331 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:29:17.331 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:29:17.617 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:29:17.618 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:29:22.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:22.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:22.619 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:29:22.659 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:29:22.659 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:29:22.936 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:29:22.937 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:29:27.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:27.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:27.938 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:29:27.978 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:29:27.978 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:29:28.247 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:29:28.248 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:29:33.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:33.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:33.249 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:29:33.290 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:29:33.290 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:29:33.591 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:29:33.591 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:29:38.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:38.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:38.592 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:29:38.635 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:29:38.635 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:29:38.887 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:29:38.887 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:29:43.890 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... FAIL 2021-02-11T01:29:44.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:44.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:44.503 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:29:44.542 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:29:44.543 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:29:44.784 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T01:29:44.829 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T01:29:44.829 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T01:29:44.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:44.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:44.830 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:29:44.869 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:29:44.869 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:29:45.135 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:29:45.136 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T01:29:45.136 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T01:29:45.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:45.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:45.136 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:29:45.175 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:29:45.175 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:29:45.438 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:29:45.439 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T01:29:45.439 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T01:29:45.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:45.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:45.439 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:29:45.480 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:29:45.480 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:29:45.743 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:29:45.744 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:29:50.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:50.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:50.745 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:29:50.786 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:29:50.786 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:29:51.065 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:29:51.066 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:29:56.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:56.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:29:56.067 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:29:56.108 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:29:56.108 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:29:56.382 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:29:56.383 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:30:01.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:01.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:01.384 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:30:01.424 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:30:01.425 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:30:01.689 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:30:01.690 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:30:06.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:06.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:06.691 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:30:06.734 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:30:06.734 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:30:06.995 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:30:06.996 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:30:11.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:11.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:11.997 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:30:12.039 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:30:12.039 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:30:12.308 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:30:12.308 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:30:17.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:17.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:17.309 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:30:17.350 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:30:17.350 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:30:17.624 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:30:17.625 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:30:22.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:22.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:22.627 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:30:22.666 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:30:22.666 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:30:22.923 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:30:22.924 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:30:27.924 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:27.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:27.925 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:30:27.968 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:30:27.968 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:30:28.236 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:30:28.237 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:30:33.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:33.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:33.238 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:30:33.280 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:30:33.281 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:30:33.551 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:30:33.551 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:30:38.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:38.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:38.552 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:30:38.592 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:30:38.593 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:30:38.858 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:30:38.859 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:30:43.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:43.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:43.860 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:30:43.903 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:30:43.903 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:30:44.179 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:30:44.179 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:30:49.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:49.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:49.181 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:30:49.219 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:30:49.219 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:30:49.504 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:30:49.505 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:30:54.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:54.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:54.506 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:30:54.548 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:30:54.548 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:30:54.810 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:30:54.811 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:30:59.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:59.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:30:59.812 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:30:59.853 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:30:59.854 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:31:00.115 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:31:00.115 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:31:05.116 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:05.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:05.117 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:31:05.155 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:31:05.155 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:31:05.427 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:31:05.428 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:31:10.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:10.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:10.429 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:31:10.471 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:31:10.471 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:31:10.753 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:31:10.754 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:31:15.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:15.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:15.755 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:31:15.796 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:31:15.796 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:31:16.061 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:31:16.062 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:31:21.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:21.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:21.063 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:31:21.109 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:31:21.109 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:31:21.362 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:31:21.362 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:31:26.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:26.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:26.529 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:31:26.570 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:31:26.570 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:31:26.827 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:31:26.827 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:31:31.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:31.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:31.828 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:31:31.874 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:31:31.874 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:31:32.135 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:31:32.135 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:31:37.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:37.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:37.136 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:31:37.174 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:31:37.174 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:31:37.427 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:31:37.428 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:31:42.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:42.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:42.429 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:31:42.471 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:31:42.471 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:31:42.725 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:31:42.725 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:31:47.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:47.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:47.726 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:31:47.768 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:31:47.769 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:31:48.038 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:31:48.039 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:31:53.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:53.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:53.040 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:31:53.087 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:31:53.088 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:31:53.351 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:31:53.351 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:31:58.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:58.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:31:58.353 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:31:58.395 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:31:58.395 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:31:58.656 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:31:58.656 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:32:03.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:03.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:03.657 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:32:03.698 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:32:03.698 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:32:03.960 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:32:03.960 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:32:08.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:08.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:08.962 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:32:09.003 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:32:09.004 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:32:09.270 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:32:09.270 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:32:14.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:14.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:14.272 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:32:14.311 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:32:14.311 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:32:14.577 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:32:14.578 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:32:19.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:19.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:19.579 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:32:19.620 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:32:19.620 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:32:19.897 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:32:19.898 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:32:24.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:24.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:24.899 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:32:24.942 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:32:24.942 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:32:25.193 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:32:25.193 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:32:30.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:30.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:30.195 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:32:30.231 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:32:30.231 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:32:30.494 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:32:30.495 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:32:35.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:35.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:35.496 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:32:35.537 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:32:35.537 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:32:35.797 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:32:35.798 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:32:40.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:40.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:40.799 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:32:40.840 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:32:40.840 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:32:41.107 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:32:41.107 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:32:46.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:46.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:46.109 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:32:46.149 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:32:46.149 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:32:46.401 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:32:46.402 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:32:51.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:51.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:51.403 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:32:51.442 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:32:51.442 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:32:51.698 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:32:51.699 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:32:56.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:56.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:32:56.700 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:32:56.741 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:32:56.742 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:32:57.004 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:32:57.004 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:33:02.005 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:02.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:02.005 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:33:02.047 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:33:02.047 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:33:02.313 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:33:02.314 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:33:07.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:07.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:07.315 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:33:07.354 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:33:07.354 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:33:07.655 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:33:07.655 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:33:12.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:12.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:12.921 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:33:12.961 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:33:12.961 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:33:13.233 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:33:13.234 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:33:18.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:18.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:18.235 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:33:18.277 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:33:18.277 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:33:18.549 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:33:18.549 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:33:23.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:23.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:23.551 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:33:23.594 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:33:23.595 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:33:23.861 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:33:23.862 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:33:28.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:28.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:28.863 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:33:28.904 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:33:28.904 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:33:29.177 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:33:29.177 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:33:34.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:34.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:34.179 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:33:34.219 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:33:34.220 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:33:34.490 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:33:34.490 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:33:39.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:39.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:39.491 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:33:39.530 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:33:39.531 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:33:39.797 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:33:39.797 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:33:44.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:44.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:44.799 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:33:44.839 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:33:44.839 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:33:45.119 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:33:45.119 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:33:50.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:50.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:50.121 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:33:50.163 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:33:50.163 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:33:50.432 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:33:50.433 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:33:55.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:55.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:33:55.434 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:33:55.476 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:33:55.476 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:33:55.756 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:33:55.756 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:34:00.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:00.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:00.758 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:34:00.800 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:34:00.800 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:34:01.063 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:34:01.063 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:34:06.064 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:06.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:06.065 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:34:06.107 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:34:06.107 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:34:06.393 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:34:06.393 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:34:11.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:13.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:13.156 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:34:13.197 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:34:13.197 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:34:13.479 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:34:13.480 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:34:18.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:18.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:18.481 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:34:18.522 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:34:18.523 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:34:18.796 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:34:18.796 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:34:23.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:23.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:23.797 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:34:23.838 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:34:23.838 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:34:24.121 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:34:24.122 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:34:29.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:29.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:29.123 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:34:29.165 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:34:29.165 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:34:29.435 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:34:29.435 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:34:34.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:34.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:34.726 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:34:34.765 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:34:34.766 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:34:35.028 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:34:35.028 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:34:40.029 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:40.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:40.030 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:34:40.070 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:34:40.070 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:34:40.338 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:34:40.339 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:34:45.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:45.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:45.340 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:34:45.381 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:34:45.382 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:34:45.658 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:34:45.659 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:34:50.659 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:50.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:50.660 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:34:50.702 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:34:50.702 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:34:50.975 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:34:50.975 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:34:55.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:56.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:34:56.064 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:34:56.107 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:34:56.107 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:34:56.393 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:34:56.393 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:35:01.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:01.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:01.395 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:35:01.434 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:35:01.434 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:35:01.696 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:35:01.696 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:35:06.720 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... FAIL 2021-02-11T01:35:06.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:06.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:06.721 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:35:06.761 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:35:06.761 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:35:06.993 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T01:35:07.037 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T01:35:07.037 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T01:35:07.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:07.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:07.038 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:35:07.079 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:35:07.079 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:35:07.331 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:35:07.331 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T01:35:07.331 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T01:35:07.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:07.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:07.332 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:35:07.373 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:35:07.373 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:35:07.636 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:35:07.636 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T01:35:07.637 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T01:35:07.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:07.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:07.637 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:35:07.679 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:35:07.679 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:35:07.935 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:35:07.936 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:35:12.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:12.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:12.937 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:35:12.977 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:35:12.977 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:35:13.233 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:35:13.234 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:35:18.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:18.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:18.236 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:35:18.278 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:35:18.278 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:35:18.556 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:35:18.556 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:35:23.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:23.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:23.558 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:35:23.598 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:35:23.598 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:35:23.876 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:35:23.876 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:35:28.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:28.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:28.878 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:35:28.920 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:35:28.920 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:35:29.197 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:35:29.198 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:35:34.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:34.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:34.199 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:35:34.238 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:35:34.238 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:35:34.512 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:35:34.513 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:35:39.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:39.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:39.514 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:35:39.555 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:35:39.555 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:35:39.830 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:35:39.831 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:35:44.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:44.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:44.832 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:35:44.873 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:35:44.873 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:35:45.138 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:35:45.138 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:35:50.139 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:50.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:50.139 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:35:50.181 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:35:50.181 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:35:50.442 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:35:50.442 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:35:55.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:55.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:35:55.444 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:35:55.486 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:35:55.486 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:35:55.764 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:35:55.764 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:36:00.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:00.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:00.765 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:36:00.807 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:36:00.807 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:36:01.088 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:36:01.088 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:36:06.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:07.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:07.503 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:36:07.542 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:36:07.542 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:36:07.813 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:36:07.813 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:36:12.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:12.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:12.815 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:36:12.858 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:36:12.858 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:36:13.128 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:36:13.128 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:36:18.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:18.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:18.129 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:36:18.179 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:36:18.179 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:36:18.447 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:36:18.448 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:36:23.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:23.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:23.450 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:36:23.491 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:36:23.491 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:36:23.772 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:36:23.772 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:36:28.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:28.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:28.773 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:36:28.813 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:36:28.814 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:36:29.085 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:36:29.086 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:36:34.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:34.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:34.088 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:36:34.129 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:36:34.129 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:36:34.411 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:36:34.412 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:36:39.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:39.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:39.413 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:36:39.454 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:36:39.454 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:36:39.718 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:36:39.719 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:36:44.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:44.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:44.720 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:36:44.761 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:36:44.761 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:36:45.050 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:36:45.050 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:36:50.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:50.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:50.052 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:36:50.094 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:36:50.094 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:36:50.361 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:36:50.361 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:36:55.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:55.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:36:55.363 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:36:55.404 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:36:55.404 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:36:55.676 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:36:55.676 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:37:00.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:00.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:00.677 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:37:00.718 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:37:00.719 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:37:00.998 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:37:00.999 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:37:06.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:06.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:06.000 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:37:06.041 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:37:06.041 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:37:06.332 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:37:06.332 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:37:11.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:11.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:11.334 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:37:11.373 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:37:11.373 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:37:11.637 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:37:11.637 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:37:16.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:16.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:16.639 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:37:16.679 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:37:16.679 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:37:16.955 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:37:16.955 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:37:21.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:21.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:21.956 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:37:21.998 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:37:21.999 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:37:22.268 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:37:22.269 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:37:27.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:27.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:27.271 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:37:27.309 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:37:27.309 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:37:27.572 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:37:27.573 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:37:32.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:32.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:32.575 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:37:32.614 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:37:32.614 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:37:32.884 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:37:32.885 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:37:37.886 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:37.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:37.886 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:37:37.926 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:37:37.926 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:37:38.191 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:37:38.191 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:37:43.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:43.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:43.192 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:37:43.228 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:37:43.228 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:37:43.479 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:37:43.479 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:37:48.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:48.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:48.481 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:37:48.522 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:37:48.522 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:37:48.802 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:37:48.803 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:37:53.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:53.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:37:53.804 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:37:53.846 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:37:53.846 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:37:54.120 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:37:55.342 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:38:00.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:00.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:00.344 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:38:00.381 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:38:00.381 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:38:00.666 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:38:00.667 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:38:05.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:05.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:05.668 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:38:05.709 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:38:05.709 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:38:05.987 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:38:05.987 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:38:10.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:10.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:10.988 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:38:11.030 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:38:11.030 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:38:11.293 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:38:11.294 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:38:16.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:16.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:16.295 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:38:16.339 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:38:16.339 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:38:16.618 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:38:16.619 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:38:21.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:21.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:21.620 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:38:21.662 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:38:21.662 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:38:21.935 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:38:21.935 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:38:26.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:26.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:26.937 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:38:26.977 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:38:26.978 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:38:27.253 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:38:27.254 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:38:32.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:32.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:32.255 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:38:32.294 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:38:32.294 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:38:32.550 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:38:32.551 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:38:37.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:37.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:37.553 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:38:37.594 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:38:37.595 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:38:37.863 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:38:37.863 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:38:42.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:42.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:42.864 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:38:42.906 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:38:42.906 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:38:43.164 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:38:43.165 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:38:48.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:48.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:48.166 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:38:48.199 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:38:48.199 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:38:48.435 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:38:48.436 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:38:53.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:53.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:53.437 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:38:53.479 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:38:53.479 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:38:53.751 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:38:53.752 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:38:58.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:58.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:38:58.754 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:38:58.793 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:38:58.794 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:38:59.083 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:38:59.083 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:39:04.084 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:04.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:04.915 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:39:04.956 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:39:04.956 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:39:05.218 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:39:05.218 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:39:10.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:10.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:10.220 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:39:10.260 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:39:10.260 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:39:10.524 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:39:10.525 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:39:15.526 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:15.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:15.526 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:39:15.568 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:39:15.568 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:39:15.829 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:39:15.830 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:39:20.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:20.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:20.831 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:39:20.873 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:39:20.873 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:39:21.129 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:39:21.130 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:39:26.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:26.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:26.132 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:39:26.172 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:39:26.172 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:39:26.444 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:39:26.445 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:39:31.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:31.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:31.446 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:39:31.486 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:39:31.486 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:39:31.755 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:39:31.755 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:39:36.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:36.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:36.756 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:39:36.797 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:39:36.797 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:39:37.061 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:39:37.062 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:39:42.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:42.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:42.063 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:39:42.103 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:39:42.103 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:39:42.373 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:39:42.373 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:39:47.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:48.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:48.479 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:39:48.520 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:39:48.520 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:39:48.780 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:39:48.781 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:39:53.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:53.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:53.782 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:39:53.822 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:39:53.822 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:39:54.089 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:39:54.090 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:39:59.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:59.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:39:59.091 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:39:59.134 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:39:59.134 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:39:59.392 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:39:59.393 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:40:04.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:04.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:04.394 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:40:04.434 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:40:04.434 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:40:04.723 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:40:04.724 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:40:09.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:09.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:09.725 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:40:09.766 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:40:09.766 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:40:10.026 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:40:10.026 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:40:15.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:15.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:15.028 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:40:15.070 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:40:15.070 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:40:15.333 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:40:15.334 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:40:20.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:20.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:20.335 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:40:20.372 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:40:20.372 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:40:20.642 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:40:20.642 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:40:25.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:25.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:25.643 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:40:25.686 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:40:25.687 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:40:25.952 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:40:25.952 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:40:30.975 INFO:tasks.rgw_multisite_tests:test s3 topics set/get/delete on master ... FAIL 2021-02-11T01:40:30.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:30.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:30.976 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:40:31.014 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:40:31.014 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:40:31.240 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T01:40:31.283 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T01:40:31.283 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T01:40:31.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:31.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:31.284 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:40:31.326 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:40:31.326 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:40:31.579 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:40:31.580 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T01:40:31.580 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T01:40:31.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:31.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:31.580 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:40:31.624 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:40:31.624 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:40:31.868 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:40:31.869 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T01:40:31.869 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T01:40:31.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:31.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:31.869 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:40:31.910 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:40:31.911 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:40:32.186 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:40:32.186 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:40:37.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:37.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:37.188 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:40:37.230 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:40:37.230 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:40:37.505 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:40:37.505 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:40:42.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:42.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:42.506 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:40:42.545 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:40:42.546 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:40:42.821 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:40:42.821 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:40:47.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:47.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:47.822 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:40:47.862 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:40:47.863 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:40:48.129 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:40:48.130 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:40:53.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:54.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:40:54.683 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:40:54.724 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:40:54.724 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:40:55.031 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:40:55.031 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:41:00.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:00.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:00.033 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:41:00.071 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:41:00.071 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:41:00.321 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:41:00.321 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:41:05.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:05.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:05.323 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:41:05.364 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:41:05.364 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:41:05.643 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:41:05.644 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:41:10.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:10.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:10.644 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:41:10.687 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:41:10.688 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:41:10.958 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:41:10.958 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:41:15.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:15.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:15.959 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:41:16.002 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:41:16.002 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:41:16.269 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:41:16.270 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:41:21.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:21.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:21.271 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:41:21.312 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:41:21.312 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:41:21.568 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:41:21.568 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:41:26.569 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:26.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:26.569 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:41:26.610 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:41:26.610 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:41:26.874 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:41:26.875 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:41:31.875 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:31.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:31.876 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:41:31.918 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:41:31.918 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:41:32.182 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:41:32.183 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:41:37.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:37.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:37.185 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:41:37.227 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:41:37.228 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:41:37.476 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:41:37.476 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:41:42.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:42.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:42.477 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:41:42.518 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:41:42.518 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:41:42.798 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:41:42.798 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:41:47.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:47.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:47.800 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:41:47.840 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:41:47.840 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:41:48.119 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:41:48.119 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:41:53.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:53.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:53.121 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:41:53.163 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:41:53.163 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:41:53.420 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:41:53.421 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:41:58.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:58.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:41:58.422 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:41:58.462 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:41:58.463 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:41:58.714 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:41:58.714 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:42:03.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:03.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:03.716 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:42:03.756 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:42:03.757 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:42:04.022 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:42:04.022 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:42:09.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:09.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:09.024 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:42:09.064 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:42:09.065 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:42:09.333 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:42:09.333 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:42:14.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:14.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:14.335 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:42:14.374 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:42:14.375 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:42:14.623 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:42:14.624 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:42:19.625 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:19.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:19.625 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:42:19.666 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:42:19.666 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:42:19.937 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:42:19.938 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:42:24.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:24.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:24.939 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:42:24.979 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:42:24.980 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:42:25.248 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:42:25.249 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:42:30.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:30.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:30.250 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:42:30.292 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:42:30.292 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:42:30.536 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:42:30.536 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:42:35.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:35.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:35.538 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:42:35.578 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:42:35.579 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:42:35.840 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:42:35.840 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:42:40.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:40.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:40.842 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:42:40.881 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:42:40.881 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:42:41.158 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:42:41.158 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:42:46.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:46.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:46.159 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:42:46.200 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:42:46.200 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:42:46.473 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:42:46.473 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:42:51.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:51.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:51.475 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:42:51.517 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:42:51.517 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:42:51.802 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:42:51.802 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:42:56.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:56.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:42:56.804 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:42:56.846 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:42:56.846 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:42:57.101 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:42:57.101 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:43:02.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:02.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:02.103 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:43:02.144 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:43:02.144 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:43:02.383 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:43:02.383 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:43:07.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:07.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:07.385 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:43:07.426 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:43:07.426 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:43:07.705 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:43:07.705 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:43:12.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:12.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:12.707 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:43:12.750 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:43:12.750 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:43:13.024 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:43:13.024 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:43:18.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:18.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:18.025 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:43:18.065 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:43:18.065 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:43:18.329 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:43:18.329 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:43:23.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:23.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:23.330 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:43:23.370 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:43:23.370 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:43:23.631 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:43:23.631 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:43:28.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:29.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:29.557 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:43:29.597 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:43:29.597 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:43:29.863 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:43:29.864 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:43:34.865 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:34.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:34.865 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:43:34.905 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:43:34.905 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:43:35.179 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:43:35.179 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:43:40.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:40.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:40.180 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:43:40.217 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:43:40.217 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:43:40.489 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:43:40.490 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:43:45.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:45.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:45.491 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:43:45.531 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:43:45.531 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:43:45.789 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:43:45.789 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:43:50.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:50.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:50.790 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:43:50.832 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:43:50.832 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:43:51.100 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:43:51.100 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:43:56.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:56.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:43:56.102 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:43:56.143 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:43:56.143 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:43:56.424 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:43:56.425 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:44:01.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:01.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:01.506 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:44:01.547 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:44:01.547 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:44:01.825 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:44:01.825 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:44:06.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:06.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:06.827 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:44:06.867 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:44:06.868 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:44:07.133 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:44:07.134 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:44:12.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:12.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:12.135 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:44:12.180 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:44:12.180 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:44:12.463 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:44:12.463 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:44:17.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:17.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:17.529 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:44:17.572 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:44:17.572 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:44:17.837 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:44:17.837 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:44:22.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:22.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:22.838 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:44:22.881 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:44:22.882 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:44:23.146 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:44:23.147 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:44:28.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:28.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:28.148 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:44:28.191 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:44:28.191 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:44:28.444 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:44:28.445 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:44:33.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:33.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:33.445 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:44:33.485 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:44:33.485 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:44:33.749 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:44:33.749 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:44:38.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:38.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:38.751 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:44:38.790 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:44:38.790 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:44:39.076 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:44:39.076 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:44:44.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:44.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:44.078 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:44:44.118 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:44:44.118 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:44:44.374 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:44:44.374 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:44:49.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:49.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:49.375 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:44:49.417 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:44:49.417 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:44:49.673 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:44:49.674 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:44:54.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:54.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:54.675 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:44:54.718 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:44:54.718 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:44:54.990 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:44:54.990 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:44:59.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:59.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:44:59.992 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:45:00.034 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:45:00.035 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:45:00.280 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:45:00.280 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:45:05.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:05.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:05.281 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:45:05.321 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:45:05.321 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:45:05.587 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:45:05.587 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:45:10.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:10.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:10.589 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:45:10.630 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:45:10.630 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:45:10.895 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:45:10.895 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:45:15.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:15.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:15.897 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:45:15.937 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:45:15.937 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:45:16.213 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:45:16.213 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:45:21.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:21.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:21.214 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:45:21.250 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:45:21.250 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:45:21.509 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:45:21.510 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:45:26.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:26.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:26.605 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:45:26.645 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:45:26.645 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:45:26.907 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:45:26.907 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:45:31.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:31.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:31.908 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:45:31.951 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:45:31.951 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:45:32.213 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:45:32.213 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:45:37.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:37.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:37.215 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:45:37.250 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:45:37.250 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:45:37.512 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:45:37.512 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:45:42.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:42.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:42.513 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:45:42.555 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:45:42.555 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:45:42.839 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:45:42.840 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:45:47.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:47.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:47.842 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:45:47.883 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:45:47.883 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:45:48.148 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:45:48.149 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:45:53.170 INFO:tasks.rgw_multisite_tests:test s3 topics with secret set/get/delete on master ... FAIL 2021-02-11T01:45:53.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:53.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:53.172 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:45:53.212 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:45:53.212 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:45:53.443 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T01:45:53.486 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T01:45:53.486 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T01:45:53.487 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:53.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:53.487 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:45:53.527 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:45:53.527 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:45:53.783 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:45:53.784 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T01:45:53.784 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T01:45:53.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:53.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:53.784 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:45:53.826 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:45:53.826 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:45:54.088 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:45:54.088 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T01:45:54.088 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T01:45:54.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:54.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:54.089 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:45:54.129 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:45:54.129 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:45:54.385 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:45:54.386 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:45:59.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:59.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:45:59.387 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:45:59.426 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:45:59.426 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:45:59.694 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:45:59.695 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:46:04.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:04.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:04.696 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:46:04.738 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:46:04.738 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:46:05.043 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:46:05.043 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:46:10.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:10.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:10.045 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:46:10.087 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:46:10.087 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:46:10.355 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:46:10.355 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:46:15.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:15.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:15.357 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:46:15.397 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:46:15.397 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:46:15.660 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:46:15.660 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:46:20.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:20.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:20.662 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:46:20.702 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:46:20.702 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:46:20.984 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:46:20.985 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:46:25.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:25.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:25.986 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:46:26.027 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:46:26.027 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:46:26.310 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:46:26.310 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:46:31.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:31.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:31.312 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:46:31.352 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:46:31.352 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:46:31.632 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:46:31.632 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:46:36.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:36.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:36.633 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:46:36.675 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:46:36.675 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:46:36.933 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:46:36.933 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:46:41.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:41.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:41.935 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:46:41.976 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:46:41.976 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:46:42.243 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:46:42.243 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:46:47.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:50.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:50.423 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:46:50.466 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:46:50.466 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:46:50.732 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:46:50.733 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:46:55.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:55.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:46:55.734 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:46:55.774 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:46:55.775 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:46:56.061 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:46:56.061 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:47:01.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:01.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:01.063 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:47:01.102 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:47:01.102 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:47:01.371 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:47:01.372 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:47:06.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:06.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:06.373 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:47:06.412 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:47:06.413 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:47:06.678 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:47:06.678 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:47:11.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:11.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:11.680 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:47:11.723 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:47:11.723 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:47:12.006 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:47:12.007 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:47:17.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:17.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:17.008 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:47:17.049 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:47:17.049 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:47:17.315 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:47:17.316 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:47:22.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:22.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:22.317 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:47:22.357 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:47:22.357 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:47:22.625 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:47:22.625 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:47:27.627 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:27.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:27.627 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:47:27.668 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:47:27.668 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:47:27.953 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:47:27.954 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:47:32.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:32.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:32.956 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:47:32.995 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:47:32.995 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:47:33.293 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:47:33.293 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:47:38.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:38.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:38.295 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:47:38.333 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:47:38.334 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:47:38.605 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:47:38.605 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:47:43.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:43.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:43.607 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:47:43.649 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:47:43.649 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:47:43.920 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:47:43.921 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:47:48.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:48.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:48.922 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:47:48.961 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:47:48.961 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:47:49.251 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:47:49.252 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:47:54.252 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:56.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:47:56.235 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:47:56.273 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:47:56.273 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:47:56.560 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:47:56.561 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:48:01.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:01.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:01.562 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:48:01.604 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:48:01.604 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:48:01.868 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:48:01.868 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:48:06.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:06.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:06.870 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:48:06.910 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:48:06.910 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:48:07.185 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:48:07.186 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:48:12.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:12.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:12.187 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:48:12.226 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:48:12.226 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:48:12.521 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:48:12.521 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:48:17.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:17.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:17.523 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:48:17.564 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:48:17.564 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:48:17.822 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:48:17.823 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:48:22.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:22.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:22.824 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:48:22.865 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:48:22.865 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:48:23.129 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:48:23.130 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:48:28.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:28.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:28.131 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:48:28.177 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:48:28.178 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:48:28.441 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:48:28.441 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:48:33.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:33.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:33.443 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:48:33.489 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:48:33.489 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:48:33.761 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:48:33.762 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:48:38.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:38.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:38.763 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:48:38.804 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:48:38.804 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:48:39.061 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:48:39.061 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:48:44.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:44.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:44.062 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:48:44.101 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:48:44.102 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:48:44.372 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:48:44.372 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:48:49.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:49.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:49.373 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:48:49.415 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:48:49.415 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:48:49.687 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:48:49.687 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:48:54.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:54.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:48:54.689 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:48:54.729 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:48:54.729 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:48:55.014 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:48:55.014 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:49:00.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:00.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:00.015 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:49:00.053 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:49:00.053 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:49:00.294 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:49:00.294 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:49:05.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:05.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:05.295 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:49:05.336 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:49:05.336 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:49:05.607 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:49:05.608 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:49:10.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:10.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:10.609 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:49:10.649 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:49:10.649 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:49:10.921 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:49:10.921 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:49:15.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:15.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:15.923 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:49:15.965 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:49:15.965 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:49:16.231 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:49:16.231 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:49:21.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:21.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:21.233 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:49:21.270 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:49:21.271 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:49:21.536 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:49:21.536 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:49:26.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:26.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:26.537 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:49:26.580 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:49:26.580 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:49:26.858 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:49:26.858 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:49:31.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:31.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:31.860 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:49:31.902 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:49:31.902 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:49:32.152 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:49:32.153 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:49:37.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:37.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:37.712 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:49:37.753 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:49:37.753 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:49:38.015 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:49:38.015 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:49:43.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:43.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:43.017 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:49:43.057 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:49:43.057 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:49:43.309 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:49:43.310 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:49:48.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:48.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:48.310 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:49:48.351 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:49:48.351 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:49:48.628 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:49:48.629 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:49:53.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:53.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:53.630 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:49:53.670 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:49:53.670 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:49:53.970 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:49:53.970 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:49:58.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:58.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:49:58.972 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:49:59.014 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:49:59.014 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:49:59.273 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:49:59.274 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:50:04.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:04.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:04.275 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:50:04.317 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:50:04.317 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:50:04.595 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:50:04.596 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:50:09.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:09.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:09.750 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:50:09.791 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:50:09.791 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:50:10.057 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:50:10.058 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:50:15.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:15.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:15.060 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:50:15.102 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:50:15.102 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:50:15.398 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:50:15.399 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:50:20.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:20.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:20.400 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:50:20.442 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:50:20.442 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:50:20.713 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:50:20.713 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:50:25.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:26.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:26.236 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:50:26.278 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:50:26.278 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:50:26.539 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:50:26.540 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:50:31.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:31.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:31.541 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:50:31.581 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:50:31.582 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:50:31.876 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:50:31.876 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:50:36.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:36.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:36.878 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:50:36.918 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:50:36.918 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:50:37.171 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:50:37.172 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:50:42.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:42.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:42.173 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:50:42.213 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:50:42.213 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:50:42.484 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:50:42.485 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:50:47.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:47.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:47.486 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:50:47.527 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:50:47.528 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:50:47.807 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:50:47.807 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:50:52.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:52.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:50:52.808 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:50:52.850 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:50:52.850 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:50:53.125 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:50:55.953 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:51:00.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:00.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:00.954 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:51:00.993 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:51:00.994 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:51:01.281 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:51:01.281 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:51:06.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:06.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:06.282 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:51:06.321 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:51:06.321 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:51:06.590 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:51:06.591 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:51:11.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:11.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:11.592 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:51:11.633 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:51:11.633 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:51:11.918 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:51:11.918 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:51:16.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:16.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:16.920 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:51:16.962 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:51:16.962 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:51:17.232 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:51:18.392 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:51:23.414 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... FAIL 2021-02-11T01:51:23.415 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter_on_master ... ok 2021-02-11T01:51:23.415 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter ... ok 2021-02-11T01:51:23.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:23.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:23.416 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:51:23.458 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:51:23.458 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:51:23.677 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T01:51:23.720 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T01:51:23.720 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T01:51:23.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:23.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:23.721 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:51:23.761 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:51:23.761 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:51:24.023 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:51:24.023 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T01:51:24.024 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T01:51:24.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:24.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:24.024 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:51:24.070 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:51:24.071 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:51:24.321 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:51:24.322 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T01:51:24.322 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T01:51:24.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:24.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:24.322 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:51:24.363 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:51:24.364 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:51:24.632 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:51:24.632 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:51:29.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:29.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:29.634 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:51:29.674 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:51:29.674 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:51:29.946 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:51:29.946 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:51:34.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:35.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:35.688 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:51:35.729 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:51:35.729 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:51:36.000 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:51:36.001 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:51:41.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:41.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:41.002 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:51:41.043 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:51:41.043 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:51:41.326 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:51:41.327 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:51:46.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:46.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:46.328 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:51:46.369 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:51:46.369 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:51:46.633 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:51:46.633 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:51:51.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:51.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:51.635 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:51:51.678 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:51:51.678 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:51:51.949 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:51:51.950 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:51:56.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:58.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:51:58.766 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:51:58.807 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:51:58.808 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:51:59.102 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:51:59.102 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:52:04.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:04.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:04.104 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:52:04.144 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:52:04.144 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:52:04.405 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:52:04.406 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:52:09.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:09.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:09.407 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:52:09.449 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:52:09.450 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:52:09.723 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:52:09.723 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:52:14.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:14.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:14.724 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:52:14.766 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:52:14.766 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:52:15.048 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:52:15.048 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:52:20.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:20.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:20.050 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:52:20.090 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:52:20.090 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:52:20.343 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:52:22.130 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:52:27.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:27.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:27.131 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:52:27.169 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:52:27.170 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:52:27.409 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:52:27.410 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:52:32.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:32.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:32.411 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:52:32.452 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:52:32.452 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:52:32.743 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:52:32.744 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:52:37.744 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:37.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:37.745 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:52:37.786 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:52:37.786 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:52:38.052 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:52:38.053 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:52:43.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:43.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:43.055 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:52:43.097 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:52:43.098 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:52:43.340 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:52:43.341 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:52:48.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:48.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:48.342 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:52:48.381 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:52:48.381 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:52:48.637 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:52:48.637 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:52:53.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:53.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:53.639 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:52:53.680 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:52:53.680 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:52:53.933 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:52:53.933 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:52:58.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:58.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:52:58.935 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:52:58.975 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:52:58.975 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:52:59.238 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:52:59.238 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:53:04.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:04.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:04.240 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:53:04.282 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:53:04.282 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:53:04.579 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:53:04.579 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:53:09.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:09.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:09.581 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:53:09.622 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:53:09.622 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:53:09.893 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:53:09.894 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:53:14.894 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:14.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:14.895 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:53:14.936 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:53:14.936 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:53:15.206 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:53:15.207 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:53:20.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:20.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:20.208 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:53:20.250 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:53:20.250 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:53:20.519 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:53:20.519 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:53:25.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:25.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:25.521 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:53:25.561 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:53:25.561 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:53:25.840 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:53:25.840 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:53:30.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:30.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:30.842 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:53:30.882 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:53:30.882 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:53:31.173 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:53:31.173 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:53:36.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:36.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:36.175 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:53:36.217 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:53:36.217 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:53:36.486 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:53:36.487 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:53:41.487 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:41.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:41.488 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:53:41.528 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:53:41.529 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:53:41.788 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:53:41.788 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:53:46.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:46.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:46.790 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:53:46.829 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:53:46.829 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:53:47.108 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:53:47.109 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:53:52.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:52.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:52.110 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:53:52.152 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:53:52.153 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:53:52.424 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:53:52.425 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:53:57.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:57.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:53:57.426 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:53:57.467 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:53:57.467 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:53:57.732 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:53:57.733 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:54:02.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:02.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:02.734 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:54:02.775 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:54:02.775 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:54:03.040 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:54:03.041 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:54:08.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:08.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:08.753 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:54:08.794 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:54:08.794 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:54:09.072 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:54:09.072 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:54:14.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:14.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:14.074 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:54:14.115 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:54:14.115 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:54:14.376 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:54:14.377 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:54:19.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:19.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:19.378 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:54:19.419 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:54:19.419 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:54:19.686 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:54:19.686 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:54:24.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:24.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:24.688 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:54:24.729 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:54:24.730 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:54:25.000 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:54:25.000 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:54:30.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:30.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:30.269 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:54:30.308 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:54:30.309 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:54:30.571 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:54:30.571 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:54:35.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:35.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:35.572 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:54:35.613 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:54:35.613 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:54:35.875 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:54:35.875 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:54:40.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:40.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:40.877 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:54:40.919 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:54:40.919 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:54:41.178 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:54:41.178 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:54:46.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:46.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:46.180 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:54:46.222 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:54:46.222 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:54:46.487 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:54:46.488 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:54:51.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:52.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:52.768 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:54:52.810 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:54:52.811 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:54:53.087 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:54:53.088 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:54:58.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:58.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:54:58.089 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:54:58.130 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:54:58.131 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:54:58.391 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:54:58.392 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:55:03.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:03.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:03.394 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:55:03.433 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:55:03.434 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:55:03.710 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:55:03.710 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:55:08.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:08.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:08.712 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:55:08.752 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:55:08.752 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:55:09.032 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:55:09.033 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:55:14.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:14.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:14.035 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:55:14.077 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:55:14.077 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:55:14.379 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:55:14.379 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:55:19.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:19.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:19.381 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:55:19.422 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:55:19.422 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:55:19.687 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:55:19.687 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:55:24.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:24.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:24.689 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:55:24.776 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:55:24.776 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:55:25.053 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:55:25.054 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:55:30.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:30.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:30.056 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:55:30.097 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:55:30.097 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:55:30.373 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:55:30.374 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:55:35.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:35.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:35.375 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:55:35.417 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:55:35.417 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:55:35.698 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:55:35.699 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:55:40.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:40.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:40.700 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:55:40.742 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:55:40.742 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:55:41.001 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:55:41.001 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:55:46.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:46.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:46.003 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:55:46.046 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:55:46.046 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:55:46.315 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:55:46.316 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:55:51.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:51.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:51.317 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:55:51.357 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:55:51.357 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:55:51.626 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:55:51.627 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:55:56.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:57.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:55:57.766 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:55:57.807 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:55:57.808 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:55:58.074 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:55:58.074 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:56:03.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:03.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:03.076 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:56:03.118 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:56:03.118 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:56:03.408 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:56:03.409 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:56:08.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:08.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:08.410 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:56:08.450 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:56:08.451 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:56:08.727 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:56:08.728 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:56:13.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:13.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:13.730 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:56:13.769 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:56:13.770 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:56:14.046 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:56:14.046 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:56:19.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:19.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:19.047 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:56:19.090 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:56:19.090 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:56:19.370 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:56:19.371 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:56:24.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:24.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:24.373 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:56:24.409 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:56:24.410 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:56:24.690 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:56:24.691 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:56:29.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:29.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:29.692 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:56:29.732 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:56:29.732 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:56:30.003 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:56:30.003 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:56:35.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:35.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:35.004 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:56:35.045 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:56:35.045 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:56:35.331 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:56:35.332 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:56:40.332 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:40.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:40.333 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:56:40.374 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:56:40.374 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:56:40.647 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:56:40.647 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:56:45.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:45.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:45.648 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:56:45.689 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:56:45.689 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:56:45.969 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:56:45.969 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:56:50.991 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... FAIL 2021-02-11T01:56:50.991 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_objcet_timing ... ok 2021-02-11T01:56:50.992 INFO:tasks.rgw_multisite_tests:test pushing amqp s3 notification on master ... ok 2021-02-11T01:56:50.992 INFO:tasks.rgw_multisite_tests:test reservation cleanup after gateway crash ... ok 2021-02-11T01:56:50.992 INFO:tasks.rgw_multisite_tests:test gateway recovery of persistent notifications ... ok 2021-02-11T01:56:50.992 INFO:tasks.rgw_multisite_tests:test pushing persistent notification via two gateways ... ok 2021-02-11T01:56:50.993 INFO:tasks.rgw_multisite_tests:test pushing persistent notification when one of the endpoints has error ... ok 2021-02-11T01:56:50.993 INFO:tasks.rgw_multisite_tests:test pushing persistent notification http ... ok 2021-02-11T01:56:50.993 INFO:tasks.rgw_multisite_tests:test pushing persistent notification amqp ... ok 2021-02-11T01:56:50.993 INFO:tasks.rgw_multisite_tests:test pushing persistent notification of large notifications ... ok 2021-02-11T01:56:50.993 INFO:tasks.rgw_multisite_tests:test pushing persistent notification pushback ... ok 2021-02-11T01:56:50.994 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2021-02-11T01:56:50.994 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2021-02-11T01:56:50.994 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl ... ok 2021-02-11T01:56:50.994 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl_sasl ... ok 2021-02-11T01:56:50.994 INFO:tasks.rgw_multisite_tests:test deletion of multiple keys on master ... ok 2021-02-11T01:56:50.995 INFO:tasks.rgw_multisite_tests:test pushing http s3 notification on master ... ok 2021-02-11T01:56:50.995 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2021-02-11T01:56:50.995 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification on master ... ok 2021-02-11T01:56:50.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:50.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:50.996 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:56:51.035 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:56:51.036 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:56:51.261 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T01:56:51.304 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T01:56:51.304 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T01:56:51.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:51.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:51.305 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:56:51.344 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:56:51.345 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:56:51.594 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:56:51.595 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T01:56:51.595 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T01:56:51.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:51.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:51.595 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:56:51.637 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:56:51.637 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:56:51.896 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:56:51.897 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T01:56:51.897 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T01:56:51.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:51.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:51.897 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:56:51.938 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:56:51.938 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:56:52.213 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:56:52.213 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:56:57.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:57.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:56:57.215 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:56:57.257 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:56:57.257 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:56:57.538 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:56:57.539 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:57:02.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:02.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:02.540 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:57:02.580 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:57:02.581 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:57:02.850 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:57:02.851 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:57:07.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:08.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:08.511 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:57:08.553 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:57:08.554 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:57:08.815 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:57:08.816 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:57:13.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:13.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:13.817 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:57:13.858 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:57:13.858 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:57:14.120 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:57:14.120 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:57:19.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:19.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:19.122 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:57:19.164 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:57:19.164 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:57:19.447 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:57:19.448 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:57:24.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:24.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:24.449 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:57:24.493 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:57:24.493 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:57:24.775 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:57:24.775 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:57:29.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:29.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:29.777 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:57:29.817 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:57:29.817 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:57:30.093 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:57:30.093 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:57:35.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:35.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:35.095 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:57:35.136 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:57:35.136 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:57:35.408 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:57:35.408 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:57:40.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:40.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:40.410 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:57:40.451 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:57:40.451 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:57:40.718 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:57:40.719 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:57:45.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:45.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:45.721 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:57:45.762 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:57:45.762 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:57:46.043 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:57:46.044 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:57:51.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:51.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:51.045 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:57:51.085 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:57:51.085 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:57:51.356 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:57:51.356 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:57:56.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:56.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:57:56.357 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:57:56.396 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:57:56.397 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:57:56.662 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:57:56.662 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:58:01.663 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:01.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:01.664 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:58:01.707 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:58:01.707 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:58:01.986 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:58:01.987 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:58:06.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:06.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:06.989 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:58:07.029 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:58:07.029 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:58:07.297 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:58:07.297 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:58:12.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:12.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:12.299 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:58:12.343 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:58:12.343 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:58:12.597 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:58:12.598 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:58:17.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:17.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:17.599 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:58:17.640 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:58:17.640 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:58:17.928 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:58:17.929 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:58:22.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:22.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:22.930 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:58:22.970 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:58:22.970 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:58:23.235 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:58:23.236 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:58:28.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:28.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:28.237 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:58:28.278 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:58:28.278 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:58:28.557 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:58:28.557 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:58:33.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:33.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:33.558 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:58:33.599 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:58:33.599 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:58:33.869 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:58:33.870 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:58:38.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:38.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:38.871 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:58:38.912 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:58:38.913 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:58:39.201 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:58:39.202 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:58:44.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:44.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:44.203 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:58:44.244 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:58:44.244 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:58:44.515 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:58:44.515 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:58:49.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:49.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:49.517 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:58:49.557 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:58:49.557 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:58:49.814 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:58:49.814 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:58:54.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:54.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:58:54.816 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:58:54.857 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:58:54.858 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:58:55.142 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:58:55.143 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:59:00.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:00.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:00.144 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:59:00.186 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:59:00.186 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:59:00.446 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:59:00.447 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:59:05.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:05.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:05.448 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:59:05.489 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:59:05.489 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:59:05.741 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:59:05.741 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:59:10.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:10.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:10.743 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:59:10.784 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:59:10.784 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:59:11.067 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:59:11.067 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:59:16.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:16.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:16.069 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:59:16.108 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:59:16.108 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:59:16.365 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:59:16.365 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:59:21.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:21.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:21.366 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:59:21.405 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:59:21.405 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:59:21.682 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:59:21.682 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:59:26.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:26.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:26.991 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:59:27.033 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:59:27.033 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:59:27.277 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:59:27.277 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:59:32.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:32.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:32.278 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:59:32.318 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:59:32.318 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:59:32.579 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:59:32.580 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:59:37.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:37.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:37.581 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:59:37.620 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:59:37.621 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:59:37.897 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:59:37.898 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:59:42.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:42.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:42.899 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:59:42.940 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:59:42.941 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:59:43.206 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:59:43.207 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:59:48.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:48.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:48.208 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:59:48.250 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:59:48.250 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:59:48.532 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:59:48.532 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:59:53.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:53.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:53.533 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:59:53.575 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:59:53.575 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:59:53.842 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:59:53.843 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T01:59:58.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:58.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T01:59:58.844 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T01:59:58.886 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T01:59:58.886 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T01:59:59.151 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T01:59:59.151 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:00:04.152 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:04.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:04.153 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:00:04.192 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:00:04.193 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:00:04.445 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:00:04.445 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:00:09.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:09.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:09.447 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:00:09.488 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:00:09.488 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:00:09.740 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:00:09.740 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:00:14.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:14.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:14.741 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:00:14.781 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:00:14.781 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:00:15.059 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:00:15.060 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:00:20.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:20.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:20.061 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:00:20.102 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:00:20.102 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:00:20.357 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:00:20.358 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:00:25.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:25.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:25.359 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:00:25.401 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:00:25.401 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:00:25.665 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:00:25.665 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:00:30.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:30.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:30.667 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:00:30.709 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:00:30.709 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:00:30.976 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:00:30.976 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:00:35.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:35.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:35.980 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:00:36.020 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:00:36.020 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:00:36.285 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:00:36.286 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:00:41.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:41.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:41.287 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:00:41.328 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:00:41.328 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:00:41.603 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:00:41.603 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:00:46.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:46.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:46.605 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:00:46.646 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:00:46.646 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:00:46.910 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:00:46.911 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:00:51.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:51.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:51.912 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:00:51.952 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:00:51.952 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:00:52.236 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:00:52.236 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:00:57.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:57.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:00:57.237 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:00:57.276 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:00:57.276 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:00:57.550 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:00:57.550 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:01:02.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:02.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:02.551 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:01:02.592 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:01:02.592 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:01:02.857 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:01:02.857 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:01:07.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:07.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:07.859 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:01:07.901 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:01:07.901 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:01:08.171 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:01:08.172 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:01:13.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:13.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:13.173 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:01:13.209 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:01:13.210 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:01:13.508 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:01:13.508 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:01:18.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:18.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:18.509 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:01:18.550 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:01:18.550 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:01:18.826 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:01:18.826 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:01:23.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:23.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:23.827 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:01:23.868 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:01:23.868 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:01:24.134 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:01:24.817 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:01:29.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:29.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:29.818 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:01:29.858 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:01:29.858 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:01:30.131 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:01:30.132 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:01:35.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:35.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:35.133 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:01:35.173 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:01:35.174 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:01:35.433 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:01:35.434 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:01:40.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:40.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:40.435 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:01:40.476 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:01:40.476 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:01:40.736 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:01:40.737 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:01:45.738 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:45.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:45.738 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:01:45.779 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:01:45.780 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:01:46.039 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:01:46.039 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:01:51.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:51.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:51.040 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:01:51.080 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:01:51.081 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:01:51.351 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:01:51.351 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:01:56.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:56.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:01:56.352 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:01:56.390 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:01:56.390 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:01:56.663 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:01:56.663 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:02:01.664 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:01.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:01.664 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:02:01.705 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:02:01.705 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:02:01.984 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:02:01.984 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:02:06.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:06.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:06.986 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:02:07.025 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:02:07.025 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:02:07.293 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:02:07.293 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:02:12.318 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... FAIL 2021-02-11T02:02:12.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:12.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:12.320 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:02:12.359 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:02:12.359 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:02:12.586 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T02:02:12.629 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T02:02:12.629 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T02:02:12.630 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:12.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:12.630 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:02:12.671 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:02:12.671 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:02:12.939 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:02:12.940 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T02:02:12.940 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T02:02:12.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:12.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:12.941 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:02:12.981 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:02:12.981 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:02:13.228 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:02:13.228 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T02:02:13.228 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T02:02:13.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:13.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:13.229 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:02:13.269 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:02:13.269 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:02:13.535 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:02:13.535 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:02:18.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:18.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:18.537 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:02:18.578 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:02:18.578 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:02:18.856 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:02:18.856 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:02:23.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:23.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:23.857 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:02:23.897 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:02:23.897 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:02:24.173 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:02:24.173 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:02:29.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:29.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:29.175 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:02:29.216 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:02:29.216 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:02:29.490 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:02:29.490 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:02:34.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:34.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:34.492 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:02:34.534 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:02:34.534 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:02:34.797 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:02:34.797 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:02:39.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:39.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:39.799 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:02:39.840 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:02:39.840 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:02:40.100 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:02:40.100 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:02:45.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:45.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:45.101 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:02:45.143 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:02:45.143 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:02:45.423 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:02:45.423 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:02:50.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:50.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:50.425 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:02:50.465 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:02:50.465 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:02:50.732 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:02:50.733 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:02:55.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:55.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:02:55.734 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:02:55.775 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:02:55.775 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:02:56.066 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:02:56.067 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:03:01.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:01.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:01.069 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:03:01.110 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:03:01.111 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:03:01.389 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:03:01.389 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:03:06.390 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:07.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:07.253 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:03:07.294 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:03:07.294 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:03:07.553 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:03:07.553 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:03:12.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:12.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:12.555 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:03:12.598 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:03:12.598 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:03:12.869 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:03:12.869 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:03:17.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:17.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:17.870 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:03:17.911 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:03:17.911 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:03:18.174 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:03:18.174 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:03:23.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:23.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:23.175 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:03:23.208 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:03:23.208 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:03:23.457 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:03:23.457 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:03:28.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:28.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:28.459 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:03:28.500 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:03:28.500 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:03:28.767 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:03:28.768 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:03:33.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:33.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:33.769 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:03:33.809 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:03:33.810 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:03:34.067 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:03:34.067 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:03:39.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:40.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:40.783 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:03:40.825 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:03:40.825 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:03:41.088 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:03:41.088 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:03:46.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:46.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:46.090 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:03:46.129 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:03:46.129 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:03:46.413 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:03:46.413 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:03:51.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:51.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:51.414 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:03:51.455 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:03:51.456 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:03:51.746 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:03:51.746 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:03:56.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:56.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:03:56.747 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:03:56.789 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:03:56.789 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:03:57.063 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:03:57.064 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:04:02.064 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:02.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:02.065 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:04:02.105 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:04:02.105 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:04:02.378 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:04:02.378 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:04:07.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:07.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:07.380 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:04:07.418 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:04:07.418 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:04:07.687 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:04:07.687 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:04:12.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:12.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:12.689 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:04:12.729 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:04:12.729 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:04:13.014 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:04:14.332 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:04:19.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:19.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:19.334 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:04:19.375 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:04:19.375 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:04:19.643 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:04:19.644 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:04:24.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:24.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:24.645 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:04:24.687 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:04:24.687 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:04:24.952 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:04:24.952 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:04:29.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:29.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:29.954 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:04:29.994 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:04:29.994 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:04:30.256 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:04:30.256 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:04:35.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:35.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:35.257 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:04:35.297 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:04:35.297 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:04:35.572 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:04:35.573 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:04:40.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:40.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:40.574 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:04:40.614 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:04:40.614 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:04:40.875 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:04:40.875 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:04:45.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:45.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:45.876 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:04:45.918 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:04:45.918 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:04:46.184 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:04:46.184 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:04:51.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:51.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:51.186 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:04:51.226 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:04:51.226 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:04:51.509 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:04:51.509 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:04:56.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:56.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:04:56.511 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:04:56.549 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:04:56.549 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:04:56.839 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:04:56.840 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:05:01.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:01.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:01.841 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:05:01.883 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:05:01.883 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:05:02.154 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:05:02.155 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:05:07.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:07.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:07.156 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:05:07.198 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:05:07.198 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:05:07.473 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:05:07.473 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:05:12.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:12.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:12.475 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:05:12.516 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:05:12.516 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:05:12.781 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:05:12.781 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:05:17.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:17.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:17.783 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:05:17.824 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:05:17.824 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:05:18.088 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:05:18.088 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:05:23.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:23.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:23.090 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:05:23.129 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:05:23.130 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:05:23.405 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:05:23.405 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:05:28.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:28.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:28.406 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:05:28.446 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:05:28.446 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:05:28.712 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:05:28.712 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:05:33.713 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:33.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:33.714 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:05:33.756 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:05:33.756 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:05:34.044 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:05:34.044 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:05:39.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:39.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:39.045 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:05:39.087 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:05:39.087 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:05:39.379 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:05:39.379 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:05:44.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:44.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:44.380 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:05:44.422 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:05:44.422 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:05:44.696 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:05:44.697 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:05:49.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:49.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:49.698 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:05:49.739 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:05:49.739 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:05:50.012 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:05:50.012 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:05:55.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:55.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:05:55.014 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:05:55.054 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:05:55.054 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:05:55.328 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:05:56.082 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:06:01.084 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:01.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:01.084 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:06:01.125 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:06:01.125 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:06:01.382 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:06:01.382 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:06:06.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:06.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:06.384 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:06:06.426 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:06:06.426 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:06:06.693 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:06:06.693 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:06:11.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:11.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:11.695 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:06:11.734 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:06:11.734 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:06:12.011 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:06:12.012 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:06:17.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:17.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:17.013 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:06:17.054 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:06:17.055 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:06:17.318 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:06:17.319 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:06:22.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:22.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:22.320 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:06:22.360 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:06:22.360 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:06:22.621 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:06:22.622 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:06:27.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:27.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:27.723 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:06:27.762 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:06:27.762 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:06:28.035 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:06:28.035 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:06:33.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:33.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:33.037 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:06:33.078 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:06:33.079 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:06:33.358 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:06:33.358 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:06:38.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:38.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:38.360 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:06:38.399 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:06:38.399 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:06:38.677 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:06:38.678 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:06:43.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:43.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:43.680 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:06:43.722 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:06:43.722 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:06:44.003 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:06:44.004 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:06:49.005 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:49.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:49.005 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:06:49.047 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:06:49.047 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:06:49.338 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:06:49.339 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:06:54.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:54.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:54.340 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:06:54.383 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:06:54.383 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:06:54.650 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:06:54.650 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:06:59.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:59.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:06:59.651 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:06:59.694 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:06:59.694 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:06:59.957 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:06:59.957 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:07:04.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:04.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:04.959 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:07:04.998 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:07:04.998 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:07:05.270 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:07:05.271 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:07:10.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:10.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:10.271 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:07:10.310 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:07:10.310 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:07:10.583 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:07:10.584 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:07:15.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:15.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:15.585 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:07:15.625 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:07:15.625 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:07:15.898 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:07:15.898 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:07:20.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:20.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:20.899 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:07:20.941 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:07:20.941 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:07:21.228 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:07:21.228 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:07:26.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:26.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:26.230 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:07:26.275 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:07:26.275 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:07:26.543 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:07:26.543 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:07:31.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:31.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:31.545 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:07:31.585 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:07:31.585 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:07:31.849 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:07:31.849 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:07:36.878 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... FAIL 2021-02-11T02:07:36.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:36.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:36.880 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:07:36.922 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:07:36.922 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:07:37.146 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T02:07:37.187 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T02:07:37.188 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T02:07:37.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:37.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:37.188 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:07:37.226 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:07:37.226 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:07:37.507 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:07:37.507 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T02:07:37.507 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T02:07:37.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:37.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:37.508 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:07:37.548 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:07:37.548 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:07:37.819 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:07:39.446 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T02:07:39.446 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T02:07:39.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:39.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:39.447 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:07:39.487 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:07:39.488 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:07:39.767 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:07:39.768 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:07:44.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:44.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:44.769 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:07:44.810 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:07:44.810 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:07:45.074 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:07:45.074 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:07:50.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:50.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:50.076 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:07:50.118 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:07:50.118 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:07:50.398 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:07:50.399 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:07:55.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:55.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:07:55.401 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:07:55.442 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:07:55.442 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:07:55.709 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:07:55.709 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:08:00.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:00.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:00.710 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:08:00.751 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:08:00.751 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:08:01.036 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:08:01.036 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:08:06.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:06.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:06.037 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:08:06.078 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:08:06.078 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:08:06.344 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:08:06.344 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:08:11.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:11.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:11.346 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:08:11.382 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:08:11.382 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:08:11.654 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:08:11.655 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:08:16.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:16.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:16.656 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:08:16.698 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:08:16.698 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:08:16.958 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:08:16.958 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:08:21.960 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:21.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:21.960 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:08:22.001 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:08:22.002 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:08:22.283 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:08:22.284 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:08:27.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:27.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:27.286 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:08:27.327 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:08:27.327 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:08:27.577 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:08:27.578 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:08:32.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:32.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:32.579 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:08:32.618 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:08:32.618 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:08:32.877 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:08:32.878 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:08:37.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:37.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:37.879 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:08:37.920 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:08:37.920 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:08:38.194 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:08:38.194 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:08:43.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:43.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:43.196 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:08:43.235 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:08:43.235 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:08:43.515 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:08:43.515 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:08:48.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:48.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:48.517 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:08:48.558 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:08:48.559 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:08:48.843 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:08:48.843 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:08:53.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:53.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:53.844 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:08:53.884 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:08:53.884 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:08:54.161 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:08:54.161 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:08:59.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:59.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:08:59.163 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:08:59.200 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:08:59.200 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:08:59.440 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:08:59.440 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:09:04.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:04.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:04.442 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:09:04.482 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:09:04.482 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:09:04.749 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:09:04.750 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:09:09.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:09.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:09.752 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:09:09.794 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:09:09.794 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:09:10.061 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:09:10.061 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:09:15.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:15.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:15.063 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:09:15.107 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:09:15.107 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:09:15.389 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:09:15.389 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:09:20.390 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:23.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:23.285 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:09:23.326 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:09:23.327 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:09:23.597 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:09:23.597 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:09:28.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:28.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:28.599 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:09:28.638 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:09:28.638 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:09:28.900 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:09:28.900 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:09:33.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:33.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:33.902 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:09:33.945 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:09:33.945 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:09:34.211 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:09:34.211 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:09:39.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:39.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:39.213 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:09:39.254 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:09:39.255 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:09:39.521 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:09:39.521 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:09:44.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:44.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:44.523 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:09:44.564 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:09:44.564 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:09:44.830 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:09:44.831 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:09:49.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:49.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:49.832 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:09:49.874 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:09:49.875 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:09:50.146 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:09:50.147 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:09:55.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:55.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:09:55.148 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:09:55.191 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:09:55.191 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:09:55.463 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:09:55.464 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:10:00.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:10:00.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:10:00.465 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:10:00.507 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:10:00.508 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:10:00.789 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:10:00.789 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:10:05.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:10:05.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:10:05.791 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:10:05.830 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:10:05.830 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:10:06.111 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:10:06.112 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:10:11.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:10:11.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:10:11.113 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:10:11.155 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:10:11.155 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:10:11.435 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:10:11.436 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:10:16.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:10:16.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:10:16.437 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:10:16.476 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:10:16.477 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:10:16.739 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:10:16.739 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:10:21.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:10:21.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:10:21.741 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:10:21.781 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:10:21.782 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:10:22.057 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:10:22.057 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:10:27.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:10:27.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:10:27.059 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:10:27.100 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:10:27.100 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:10:27.383 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:10:27.383 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:10:32.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:10:32.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:10:32.385 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:10:32.425 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:10:32.425 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:10:32.696 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:10:32.696 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:10:37.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:10:37.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:10:37.697 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:10:37.738 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:10:37.738 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:10:37.999 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:10:37.999 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:10:43.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:08.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:08.612 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:11:08.655 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:11:08.655 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:11:08.921 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:11:08.922 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:11:13.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:13.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:13.924 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:11:13.963 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:11:13.963 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:11:14.244 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:11:14.245 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:11:19.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:19.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:19.246 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:11:19.283 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:11:19.283 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:11:19.551 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:11:19.551 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:11:24.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:24.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:24.553 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:11:24.593 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:11:24.594 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:11:24.870 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:11:24.870 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:11:29.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:29.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:29.872 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:11:29.913 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:11:29.913 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:11:30.212 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:11:30.213 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:11:35.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:35.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:35.214 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:11:35.255 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:11:35.255 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:11:35.523 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:11:35.524 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:11:40.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:40.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:40.525 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:11:40.565 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:11:40.566 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:11:40.853 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:11:40.853 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:11:45.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:45.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:45.855 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:11:45.896 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:11:45.896 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:11:46.170 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:11:46.170 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:11:51.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:51.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:51.172 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:11:51.211 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:11:51.211 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:11:51.465 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:11:51.465 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:11:56.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:56.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:11:56.467 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:11:56.508 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:11:56.508 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:11:56.801 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:11:56.802 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:12:01.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:01.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:01.804 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:12:01.846 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:12:01.847 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:12:02.103 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:12:02.103 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:12:07.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:07.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:07.105 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:12:07.147 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:12:07.147 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:12:07.383 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:12:07.383 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:12:12.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:12.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:12.385 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:12:12.427 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:12:12.427 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:12:12.703 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:12:12.704 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:12:17.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:17.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:17.705 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:12:17.747 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:12:17.748 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:12:18.039 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:12:18.039 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:12:23.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:23.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:23.040 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:12:23.082 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:12:23.082 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:12:23.327 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:12:23.328 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:12:28.328 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:28.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:28.329 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:12:28.370 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:12:28.370 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:12:28.654 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:12:28.654 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:12:33.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:33.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:33.656 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:12:33.699 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:12:33.699 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:12:33.988 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:12:33.988 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:12:38.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:38.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:38.990 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:12:39.031 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:12:39.032 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:12:39.280 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:12:39.280 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:12:44.281 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:44.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:44.281 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:12:44.324 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:12:44.324 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:12:44.587 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:12:44.588 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:12:49.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:49.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:49.589 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:12:49.631 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:12:49.631 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:12:49.918 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:12:49.918 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:12:54.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:54.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:12:54.919 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:12:54.961 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:12:54.961 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:12:55.234 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:12:55.234 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:13:00.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:00.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:00.235 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:13:00.275 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:13:00.276 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:13:00.543 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:13:00.543 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:13:05.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:05.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:05.545 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:13:05.586 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:13:05.586 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:13:05.851 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:13:05.852 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:13:10.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:10.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:10.852 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:13:10.893 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:13:10.893 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:13:11.158 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:13:11.159 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:13:16.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:16.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:16.160 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:13:16.203 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:13:16.203 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:13:16.467 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:13:16.468 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:13:21.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:21.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:21.469 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:13:21.510 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:13:21.510 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:13:21.776 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:13:21.777 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:13:26.779 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... FAIL 2021-02-11T02:13:26.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:26.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:26.780 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:13:26.822 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:13:26.822 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:13:27.048 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T02:13:27.090 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T02:13:27.090 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T02:13:27.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:27.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:27.091 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:13:27.132 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:13:27.133 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:13:27.388 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:13:27.389 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T02:13:27.389 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T02:13:27.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:27.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:27.390 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:13:27.430 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:13:27.431 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:13:27.705 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:13:27.705 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T02:13:27.705 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T02:13:27.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:27.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:27.706 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:13:27.747 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:13:27.747 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:13:28.018 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:13:28.018 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:13:33.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:33.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:33.020 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:13:33.059 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:13:33.059 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:13:33.326 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:13:33.327 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:13:38.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:38.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:38.328 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:13:38.369 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:13:38.369 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:13:38.642 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:13:38.643 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:13:43.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:43.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:43.644 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:13:43.687 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:13:43.687 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:13:43.963 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:13:43.964 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:13:48.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:48.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:48.965 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:13:49.008 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:13:49.008 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:13:49.279 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:13:49.280 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:13:54.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:54.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:54.281 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:13:54.322 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:13:54.322 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:13:54.583 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:13:54.583 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:13:59.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:59.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:13:59.584 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:13:59.627 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:13:59.627 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:13:59.887 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:13:59.887 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:14:04.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:04.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:04.889 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:14:04.931 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:14:04.931 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:14:05.197 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:14:05.197 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:14:10.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:10.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:10.198 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:14:10.238 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:14:10.238 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:14:10.503 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:14:10.504 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:14:15.505 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:15.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:15.506 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:14:15.547 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:14:15.547 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:14:15.819 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:14:15.820 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:14:20.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:20.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:20.821 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:14:20.862 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:14:20.862 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:14:21.127 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:14:21.128 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:14:26.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:26.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:26.130 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:14:26.172 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:14:26.172 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:14:26.445 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:14:26.446 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:14:31.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:31.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:31.448 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:14:31.489 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:14:31.489 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:14:31.768 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:14:31.769 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:14:36.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:36.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:36.770 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:14:36.812 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:14:36.812 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:14:37.075 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:14:37.075 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:14:42.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:42.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:42.077 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:14:42.118 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:14:42.118 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:14:42.382 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:14:42.383 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:14:47.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:47.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:47.384 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:14:47.425 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:14:47.425 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:14:47.702 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:14:47.703 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:14:52.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:52.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:52.704 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:14:52.746 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:14:52.746 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:14:53.025 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:14:53.026 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:14:58.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:58.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:14:58.027 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:14:58.067 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:14:58.067 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:14:58.335 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:14:58.584 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:15:03.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:03.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:03.585 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:15:03.627 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:15:03.627 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:15:03.911 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:15:03.911 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:15:08.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:08.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:08.912 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:15:08.953 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:15:08.954 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:15:09.223 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:15:09.223 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:15:14.224 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:14.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:14.225 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:15:14.267 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:15:14.268 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:15:14.531 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:15:14.531 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:15:19.532 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:19.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:19.532 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:15:19.575 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:15:19.575 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:15:19.848 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:15:19.849 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:15:24.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:24.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:24.851 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:15:24.892 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:15:24.892 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:15:25.156 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:15:25.157 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:15:30.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:30.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:30.158 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:15:30.202 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:15:30.203 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:15:30.462 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:15:30.463 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:15:35.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:35.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:35.464 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:15:35.507 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:15:35.507 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:15:35.773 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:15:35.773 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:15:40.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:40.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:40.775 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:15:40.816 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:15:40.816 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:15:41.097 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:15:41.098 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:15:46.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:46.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:46.098 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:15:46.138 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:15:46.139 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:15:46.423 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:15:46.423 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:15:51.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:51.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:51.424 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:15:51.466 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:15:51.466 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:15:51.734 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:15:51.734 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:15:56.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:56.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:15:56.735 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:15:56.775 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:15:56.775 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:15:57.055 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:15:57.055 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:16:02.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:02.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:02.057 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:16:02.097 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:16:02.098 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:16:02.370 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:16:02.371 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:16:07.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:07.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:07.372 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:16:07.415 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:16:07.415 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:16:07.673 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:16:07.673 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:16:12.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:12.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:12.674 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:16:12.714 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:16:12.714 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:16:12.997 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:16:12.997 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:16:17.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:17.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:17.998 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:16:18.038 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:16:18.039 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:16:18.293 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:16:18.293 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:16:23.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:23.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:23.295 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:16:23.335 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:16:23.336 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:16:23.598 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:16:23.598 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:16:28.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:28.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:28.600 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:16:28.640 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:16:28.641 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:16:28.916 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:16:28.916 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:16:33.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:33.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:33.917 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:16:33.958 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:16:33.958 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:16:34.236 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:16:34.237 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:16:39.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:39.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:39.239 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:16:39.273 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:16:39.273 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:16:39.533 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:16:39.534 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:16:44.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:44.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:44.535 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:16:44.577 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:16:44.577 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:16:44.854 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:16:44.854 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:16:49.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:49.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:49.856 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:16:49.895 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:16:49.896 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:16:50.165 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:16:50.166 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:16:55.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:55.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:16:55.169 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:16:55.216 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:16:55.217 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:16:55.461 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:16:55.462 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:17:00.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:00.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:00.463 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:17:00.504 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:17:00.505 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:17:00.785 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:17:00.786 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:17:05.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:05.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:05.787 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:17:05.830 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:17:05.830 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:17:06.084 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:17:06.085 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:17:11.086 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:11.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:11.086 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:17:11.127 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:17:11.128 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:17:11.386 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:17:11.386 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:17:16.387 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:16.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:16.388 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:17:16.427 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:17:16.428 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:17:16.703 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:17:16.802 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:17:21.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:21.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:21.803 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:17:21.843 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:17:21.843 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:17:22.136 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:17:22.137 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:17:27.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:27.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:27.138 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:17:27.180 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:17:27.180 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:17:27.434 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:17:27.434 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:17:32.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:32.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:32.436 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:17:32.477 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:17:32.478 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:17:32.758 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:17:32.758 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:17:37.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:37.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:37.760 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:17:37.799 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:17:37.799 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:17:38.072 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:17:38.073 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:17:43.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:43.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:43.074 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:17:43.118 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:17:43.118 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:17:43.379 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:17:43.380 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:17:48.381 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:48.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:48.786 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:17:48.826 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:17:48.826 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:17:49.086 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:17:49.087 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:17:54.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:54.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:54.088 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:17:54.126 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:17:54.126 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:17:54.374 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:17:54.374 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:17:59.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:59.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:17:59.376 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:17:59.418 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:17:59.419 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:17:59.695 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:17:59.695 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:18:04.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:04.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:04.696 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:18:04.739 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:18:04.739 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:18:05.001 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:18:05.001 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:18:10.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:10.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:10.003 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:18:10.043 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:18:10.043 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:18:10.308 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:18:10.309 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:18:15.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:15.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:15.310 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:18:15.353 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:18:15.354 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:18:15.618 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:18:15.618 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:18:20.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:20.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:20.620 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:18:20.662 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:18:20.662 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:18:20.936 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:18:20.936 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:18:25.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:25.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:25.938 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:18:25.980 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:18:25.980 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:18:26.241 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:18:26.241 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:18:31.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:31.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:31.243 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:18:31.280 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:18:31.280 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:18:31.535 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:18:31.535 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:18:36.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:36.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:36.537 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:18:36.578 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:18:36.578 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:18:36.847 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:18:36.847 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:18:41.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:41.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:41.848 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:18:41.891 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:18:41.891 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:18:42.159 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:18:42.160 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:18:47.161 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... FAIL 2021-02-11T02:18:47.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:47.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:47.163 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:18:47.203 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:18:47.203 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:18:47.438 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T02:18:47.479 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T02:18:47.479 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T02:18:47.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:47.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:47.480 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:18:47.521 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:18:47.521 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:18:47.786 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:18:47.786 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T02:18:47.786 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T02:18:47.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:47.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:47.787 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:18:47.829 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:18:47.829 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:18:48.102 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:18:48.103 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T02:18:48.103 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T02:18:48.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:48.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:48.103 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:18:48.143 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:18:48.143 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:18:48.419 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:18:48.420 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:18:53.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:53.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:53.421 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:18:53.463 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:18:53.463 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:18:53.732 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:18:53.732 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:18:58.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:58.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:18:58.734 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:18:58.776 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:18:58.776 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:18:59.043 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:18:59.345 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:19:04.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:04.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:04.346 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:19:04.389 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:19:04.389 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:19:04.649 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:19:04.649 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:19:09.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:09.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:09.651 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:19:09.693 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:19:09.693 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:19:09.975 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:19:09.976 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:19:14.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:14.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:14.977 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:19:15.017 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:19:15.017 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:19:15.294 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:19:15.295 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:19:20.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:20.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:20.297 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:19:20.336 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:19:20.336 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:19:20.602 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:19:20.603 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:19:25.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:25.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:25.604 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:19:25.645 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:19:25.646 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:19:25.924 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:19:25.924 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:19:30.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:30.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:30.926 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:19:30.966 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:19:30.967 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:19:31.253 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:19:31.254 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:19:36.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:36.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:36.255 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:19:36.290 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:19:36.290 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:19:36.549 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:19:36.549 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:19:41.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:41.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:41.550 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:19:41.593 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:19:41.593 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:19:41.857 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:19:41.857 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:19:46.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:46.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:46.859 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:19:46.901 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:19:46.901 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:19:47.170 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:19:47.170 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:19:52.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:52.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:52.172 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:19:52.213 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:19:52.213 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:19:52.460 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:19:52.460 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:19:57.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:57.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:19:57.462 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:19:57.503 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:19:57.503 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:19:57.765 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:19:57.765 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:20:02.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:02.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:02.767 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:20:02.808 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:20:02.808 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:20:03.090 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:20:03.091 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:20:08.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:08.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:08.093 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:20:08.135 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:20:08.135 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:20:08.391 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:20:08.391 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:20:13.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:13.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:13.392 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:20:13.433 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:20:13.433 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:20:13.701 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:20:13.701 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:20:18.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:18.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:18.703 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:20:18.746 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:20:18.746 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:20:19.014 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:20:19.015 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:20:24.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:24.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:24.016 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:20:24.059 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:20:24.059 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:20:24.360 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:20:24.361 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:20:29.362 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:29.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:29.362 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:20:29.403 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:20:29.403 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:20:29.671 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:20:29.671 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:20:34.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:34.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:34.673 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:20:34.714 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:20:34.714 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:20:34.985 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:20:34.985 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:20:39.986 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:39.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:39.987 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:20:40.027 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:20:40.027 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:20:40.302 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:20:40.303 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:20:45.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:45.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:45.304 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:20:45.344 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:20:45.344 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:20:45.625 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:20:45.625 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:20:50.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:50.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:50.627 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:20:50.666 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:20:50.666 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:20:50.954 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:20:50.954 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:20:55.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:55.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:20:55.955 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:20:55.995 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:20:55.995 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:20:56.265 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:20:56.266 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:21:01.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:01.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:01.267 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:21:01.307 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:21:01.307 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:21:01.586 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:21:01.587 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:21:06.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:06.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:06.589 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:21:06.631 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:21:06.631 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:21:06.921 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:21:06.922 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:21:11.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:11.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:11.923 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:21:11.964 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:21:11.964 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:21:12.215 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:21:12.215 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:21:17.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:18.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:18.594 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:21:18.636 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:21:18.636 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:21:18.904 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:21:18.904 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:21:23.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:23.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:23.906 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:21:23.947 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:21:23.947 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:21:24.219 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:21:24.220 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:21:29.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:29.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:29.221 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:21:29.259 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:21:29.259 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:21:29.521 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:21:29.521 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:21:34.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:34.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:34.523 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:21:34.562 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:21:34.562 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:21:34.841 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:21:34.841 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:21:39.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:39.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:39.843 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:21:39.882 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:21:39.882 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:21:40.161 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:21:40.162 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:21:45.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:45.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:45.163 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:21:45.205 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:21:45.205 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:21:45.465 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:21:45.465 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:21:50.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:50.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:50.467 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:21:50.507 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:21:50.507 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:21:50.773 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:21:50.774 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:21:55.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:55.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:21:55.776 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:21:55.816 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:21:55.816 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:21:56.077 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:21:56.077 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:22:01.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:01.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:01.078 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:22:01.120 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:22:01.120 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:22:01.387 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:22:01.387 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:22:06.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:06.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:06.388 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:22:06.429 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:22:06.429 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:22:06.715 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:22:06.716 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:22:11.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:11.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:11.717 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:22:11.756 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:22:11.756 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:22:12.049 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:22:12.049 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:22:17.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:17.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:17.051 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:22:17.091 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:22:17.091 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:22:17.359 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:22:17.360 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:22:22.360 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:22.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:22.361 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:22:22.396 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:22:22.396 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:22:22.685 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:22:22.686 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:22:27.687 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:27.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:27.687 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:22:27.729 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:22:27.729 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:22:28.003 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:22:28.003 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:22:33.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:33.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:33.005 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:22:33.047 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:22:33.047 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:22:33.342 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:22:33.342 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:22:38.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:38.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:38.344 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:22:38.386 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:22:38.386 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:22:38.661 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:22:38.662 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:22:43.663 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:43.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:43.663 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:22:43.704 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:22:43.705 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:22:43.979 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:22:43.979 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:22:48.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:48.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:48.980 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:22:49.021 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:22:49.022 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:22:49.279 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:22:49.280 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:22:54.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:54.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:22:54.281 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:22:54.314 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:22:54.315 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:22:54.591 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:22:54.592 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:22:59.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:02.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:02.305 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:23:02.346 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:23:02.346 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:23:02.613 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:23:02.614 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:23:07.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:07.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:07.616 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:23:07.657 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:23:07.657 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:23:07.924 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:23:07.925 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:23:12.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:12.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:12.927 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:23:12.968 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:23:12.968 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:23:13.248 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:23:13.248 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:23:18.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:18.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:18.250 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:23:18.290 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:23:18.290 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:23:18.546 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:23:18.547 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:23:23.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:23.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:23.548 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:23:23.591 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:23:23.591 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:23:23.874 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:23:23.875 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:23:28.875 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:28.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:28.876 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:23:28.917 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:23:28.917 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:23:29.188 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:23:29.189 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:23:34.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:34.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:34.190 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:23:34.232 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:23:34.232 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:23:34.489 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:23:34.489 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:23:39.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:39.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:39.491 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:23:39.531 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:23:39.531 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:23:39.801 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:23:39.801 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:23:44.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:44.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:44.803 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:23:44.843 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:23:44.843 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:23:45.115 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:23:45.117 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:23:50.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:50.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:50.120 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:23:50.162 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:23:50.162 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:23:50.425 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:23:50.426 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:23:55.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:55.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:23:55.428 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:23:55.467 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:23:55.467 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:23:55.729 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:23:55.729 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:24:00.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:00.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:00.731 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:24:00.771 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:24:00.771 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:24:01.058 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:24:01.059 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:24:06.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:06.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:06.060 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:24:06.101 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:24:06.902 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:24:06.903 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:24:06.903 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:24:11.906 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... FAIL 2021-02-11T02:24:11.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:11.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:11.907 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:24:11.949 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:24:11.949 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:24:12.163 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T02:24:12.203 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T02:24:12.203 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T02:24:12.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:12.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:12.203 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:24:12.247 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:24:12.247 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:24:12.502 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:24:12.502 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T02:24:12.502 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T02:24:12.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:12.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:12.503 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:24:12.543 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:24:12.543 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:24:12.836 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:24:12.836 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T02:24:12.836 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T02:24:12.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:12.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:12.837 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:24:12.876 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:24:12.877 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:24:13.148 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:24:13.149 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:24:18.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:18.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:18.150 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:24:18.190 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:24:18.190 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:24:18.447 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:24:18.447 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:24:23.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:23.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:23.448 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:24:23.490 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:24:23.490 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:24:23.763 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:24:23.763 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:24:28.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:28.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:28.764 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:24:28.805 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:24:28.806 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:24:29.073 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:24:29.074 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:24:34.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:34.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:34.076 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:24:34.114 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:24:34.115 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:24:34.384 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:24:34.385 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:24:39.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:39.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:39.386 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:24:39.425 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:24:39.425 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:24:39.710 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:24:39.710 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:24:44.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:44.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:44.711 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:24:44.751 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:24:44.751 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:24:45.011 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:24:45.011 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:24:50.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:50.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:50.013 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:24:50.054 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:24:50.054 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:24:50.328 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:24:50.328 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:24:55.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:55.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:24:55.330 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:24:55.371 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:24:55.372 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:24:55.643 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:24:55.643 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:25:00.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:00.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:00.645 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:25:00.686 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:25:00.686 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:25:00.953 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:25:00.954 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:25:05.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:05.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:05.956 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:25:05.996 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:25:05.996 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:25:06.274 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:25:06.274 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:25:11.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:11.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:11.276 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:25:11.313 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:25:11.313 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:25:11.587 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:25:11.587 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:25:16.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:16.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:16.589 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:25:16.631 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:25:16.631 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:25:16.887 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:25:16.887 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:25:21.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:21.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:21.889 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:25:21.931 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:25:21.931 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:25:22.210 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:25:22.210 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:25:27.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:27.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:27.212 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:25:27.256 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:25:27.256 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:25:27.523 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:25:27.524 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:25:32.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:32.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:32.526 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:25:32.568 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:25:32.568 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:25:32.838 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:25:32.838 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:25:37.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:37.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:37.839 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:25:37.879 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:25:37.880 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:25:38.151 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:25:38.152 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:25:43.152 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:43.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:43.152 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:25:43.193 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:25:43.193 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:25:43.470 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:25:43.471 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:25:48.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:48.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:48.472 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:25:48.513 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:25:49.270 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:25:49.272 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:25:49.272 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:25:54.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:54.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:54.273 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:25:54.314 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:25:54.314 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:25:54.585 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:25:54.585 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:25:59.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:59.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:25:59.587 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:25:59.629 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:25:59.629 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:25:59.904 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:25:59.905 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:26:04.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:04.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:04.906 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:26:04.946 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:26:04.946 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:26:05.206 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:26:05.206 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:26:10.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:10.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:10.208 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:26:10.248 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:26:10.248 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:26:10.520 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:26:10.520 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:26:15.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:15.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:15.522 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:26:15.562 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:26:15.562 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:26:15.824 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:26:15.825 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:26:20.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:20.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:20.826 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:26:20.867 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:26:20.868 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:26:21.170 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:26:21.171 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:26:26.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:26.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:26.172 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:26:26.213 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:26:26.214 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:26:26.470 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:26:26.470 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:26:31.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:31.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:31.471 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:26:31.514 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:26:31.514 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:26:31.781 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:26:31.781 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:26:36.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:36.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:36.783 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:26:36.822 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:26:36.823 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:26:37.104 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:26:37.104 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:26:42.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:42.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:42.106 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:26:42.146 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:26:42.146 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:26:42.419 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:26:42.419 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:26:47.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:47.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:47.421 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:26:47.459 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:26:47.460 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:26:47.727 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:26:47.728 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:26:52.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:52.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:52.729 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:26:52.770 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:26:52.770 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:26:53.036 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:26:53.036 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:26:58.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:58.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:26:58.038 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:26:58.080 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:26:58.080 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:26:58.364 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:26:58.365 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:27:03.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:03.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:03.366 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:27:03.408 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:27:03.408 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:27:03.660 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:27:03.661 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:27:08.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:08.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:08.662 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:27:08.705 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:27:08.705 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:27:08.978 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:27:08.978 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:27:13.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:13.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:13.980 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:27:14.018 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:27:14.019 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:27:14.294 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:27:14.295 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:27:19.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:19.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:19.296 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:27:19.337 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:27:19.337 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:27:19.627 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:27:19.628 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:27:24.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:24.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:24.629 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:27:24.671 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:27:24.671 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:27:24.932 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:27:24.932 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:27:29.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:29.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:29.933 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:27:29.974 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:27:29.974 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:27:30.238 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:27:30.239 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:27:35.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:35.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:35.240 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:27:35.280 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:27:35.280 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:27:35.569 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:27:35.569 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:27:40.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:40.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:40.571 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:27:40.610 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:27:40.611 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:27:40.874 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:27:40.875 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:27:45.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:45.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:45.876 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:27:45.918 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:27:45.919 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:27:46.183 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:27:46.183 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:27:51.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:51.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:51.185 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:27:51.226 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:27:51.226 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:27:51.474 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:27:51.475 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:27:56.476 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:56.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:27:56.476 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:27:56.515 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:27:56.515 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:27:56.777 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:27:56.778 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:28:01.779 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:01.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:01.779 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:28:01.820 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:28:01.820 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:28:02.093 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:28:02.093 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:28:07.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:07.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:07.756 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:28:07.799 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:28:07.799 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:28:08.066 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:28:08.067 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:28:13.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:13.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:13.639 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:28:13.678 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:28:13.678 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:28:13.961 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:28:13.961 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:28:18.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:18.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:18.963 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:28:19.002 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:28:19.003 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:28:19.284 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:28:19.284 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:28:24.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:24.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:24.286 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:28:24.323 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:28:24.323 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:28:24.593 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:28:24.593 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:28:29.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:29.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:29.595 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:28:29.637 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:28:29.637 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:28:29.908 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:28:29.908 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:28:34.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:34.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:34.909 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:28:34.955 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:28:34.955 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:28:35.212 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:28:35.212 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:28:40.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:40.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:40.213 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:28:40.263 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:28:40.264 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:28:40.509 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:28:40.509 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:28:45.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:45.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:45.511 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:28:45.552 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:28:45.552 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:28:45.828 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:28:45.829 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:28:50.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:50.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:50.831 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:28:50.872 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:28:50.872 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:28:51.134 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:28:51.135 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:28:56.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:56.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:28:56.136 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:28:56.175 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:28:56.175 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:28:56.458 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:28:56.458 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:29:01.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:01.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:01.460 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:29:01.497 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:29:01.498 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:29:01.773 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:29:01.773 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:29:06.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:06.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:06.774 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:29:06.814 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:29:06.815 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:29:07.085 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:29:07.085 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:29:12.086 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:12.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:12.087 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:29:12.126 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:29:12.127 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:29:12.389 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:29:12.389 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:29:17.390 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:17.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:17.391 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:29:17.432 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:29:17.432 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:29:17.713 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:29:17.713 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:29:22.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:22.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:22.714 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:29:22.755 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:29:22.756 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:29:23.064 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:29:23.064 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:29:28.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:28.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:28.065 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:29:28.106 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:29:28.106 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:29:28.368 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:29:28.368 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:29:33.390 INFO:tasks.rgw_multisite_tests:test radosgw-admin commands ... FAIL 2021-02-11T02:29:33.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:33.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:33.392 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:29:33.431 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:29:33.431 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:29:33.679 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T02:29:33.722 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T02:29:33.722 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T02:29:33.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:33.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:33.723 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:29:33.762 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:29:33.762 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:29:34.042 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:29:34.042 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T02:29:34.042 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T02:29:34.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:34.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:34.043 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:29:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:29:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:29:34.359 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:29:34.360 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T02:29:34.360 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T02:29:34.360 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:34.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:34.360 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:29:34.399 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:29:34.399 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:29:34.660 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:29:34.660 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:29:39.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:39.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:39.661 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:29:39.703 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:29:39.703 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:29:39.975 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:29:39.975 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:29:44.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:44.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:44.977 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:29:45.019 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:29:45.019 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:29:45.290 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:29:45.290 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:29:50.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:50.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:50.292 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:29:50.327 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:29:50.327 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:29:50.583 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:29:50.584 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:29:55.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:55.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:29:55.585 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:29:55.628 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:29:55.628 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:29:55.892 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:29:55.892 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:30:00.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:00.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:00.894 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:30:00.934 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:30:00.934 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:30:01.212 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:30:01.212 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:30:06.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:06.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:06.214 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:30:06.254 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:30:06.254 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:30:06.509 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:30:06.509 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:30:11.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:11.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:11.511 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:30:11.550 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:30:11.551 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:30:11.817 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:30:11.818 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:30:16.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:16.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:16.819 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:30:16.860 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:30:16.860 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:30:17.135 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:30:17.135 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:30:22.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:22.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:22.137 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:30:22.178 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:30:22.178 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:30:22.437 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:30:22.438 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:30:27.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:27.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:27.439 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:30:27.479 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:30:27.479 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:30:27.770 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:30:27.771 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:30:32.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:32.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:32.772 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:30:32.813 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:30:32.813 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:30:33.079 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:30:33.079 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:30:38.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:38.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:38.080 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:30:38.122 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:30:38.122 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:30:38.374 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:30:38.374 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:30:43.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:43.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:43.376 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:30:43.417 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:30:43.417 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:30:43.683 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:30:43.684 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:30:48.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:48.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:48.685 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:30:48.725 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:30:48.725 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:30:48.997 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:30:48.998 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:30:53.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:53.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:53.999 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:30:54.041 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:30:54.042 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:30:54.308 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:30:54.308 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:30:59.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:59.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:30:59.310 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:30:59.350 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:30:59.351 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:30:59.625 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:30:59.625 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:31:04.627 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:04.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:04.627 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:31:04.670 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:31:04.671 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:31:04.941 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:31:04.941 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:31:09.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:09.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:09.943 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:31:09.982 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:31:09.982 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:31:10.253 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:31:10.254 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:31:15.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:15.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:15.255 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:31:15.296 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:31:15.296 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:31:15.582 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:31:15.582 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:31:20.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:20.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:20.583 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:31:20.624 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:31:20.624 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:31:20.888 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:31:20.889 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:31:25.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:25.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:25.890 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:31:25.930 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:31:25.930 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:31:26.204 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:31:26.204 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:31:31.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:31.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:31.206 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:31:31.246 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:31:31.246 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:31:31.539 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:31:31.539 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:31:36.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:36.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:36.540 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:31:36.581 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:31:36.582 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:31:36.855 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:31:36.855 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:31:41.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:41.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:41.857 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:31:41.897 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:31:41.897 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:31:42.170 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:31:42.170 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:31:47.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:48.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:48.604 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:31:48.645 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:31:48.645 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:31:48.922 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:31:48.922 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:31:53.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:53.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:53.924 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:31:53.965 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:31:53.965 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:31:54.235 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:31:54.236 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:31:59.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:59.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:31:59.238 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:31:59.276 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:31:59.276 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:31:59.521 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:31:59.522 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:32:04.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:04.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:04.523 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:32:04.563 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:32:04.563 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:32:04.844 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:32:04.844 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:32:09.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:09.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:09.846 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:32:09.886 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:32:09.886 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:32:10.167 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:32:10.167 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:32:15.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:15.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:15.169 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:32:15.210 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:32:15.210 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:32:15.463 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:32:15.464 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:32:20.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:20.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:20.466 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:32:20.504 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:32:20.504 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:32:20.776 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:32:20.776 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:32:25.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:25.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:25.777 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:32:25.819 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:32:25.819 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:32:26.093 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:32:26.093 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:32:31.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:31.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:31.095 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:32:31.136 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:32:31.136 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:32:31.397 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:32:31.398 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:32:36.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:36.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:36.399 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:32:36.435 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:32:36.435 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:32:36.693 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:32:36.693 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:32:41.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:41.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:41.695 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:32:41.737 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:32:41.737 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:32:42.007 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:32:42.008 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:32:47.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:47.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:47.010 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:32:47.051 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:32:47.051 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:32:47.319 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:32:47.320 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:32:52.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:52.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:52.321 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:32:52.362 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:32:52.362 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:32:52.606 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:32:52.607 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:32:57.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:57.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:32:57.608 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:32:57.651 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:32:57.651 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:32:57.924 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:32:57.925 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:33:02.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:02.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:02.927 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:33:02.968 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:33:02.969 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:33:03.251 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:33:03.251 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:33:08.252 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:08.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:08.253 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:33:08.293 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:33:08.294 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:33:08.539 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:33:08.540 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:33:13.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:13.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:13.541 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:33:13.584 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:33:13.584 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:33:13.857 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:33:13.857 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:33:18.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:18.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:18.859 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:33:18.900 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:33:18.900 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:33:19.168 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:33:19.169 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:33:24.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:24.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:24.171 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:33:24.215 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:33:24.216 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:33:24.471 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:33:24.471 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:33:29.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:29.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:29.777 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:33:29.817 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:33:29.818 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:33:30.076 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:33:30.076 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:33:35.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:35.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:35.078 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:33:35.120 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:33:35.120 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:33:35.357 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:33:35.358 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:33:40.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:40.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:40.359 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:33:40.402 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:33:40.403 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:33:40.656 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:33:40.656 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:33:45.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:45.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:45.658 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:33:45.698 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:33:45.699 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:33:45.973 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:33:45.973 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:33:50.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:50.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:50.974 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:33:51.014 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:33:51.014 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:33:51.276 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:33:51.276 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:33:56.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:56.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:33:56.278 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:33:56.318 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:33:56.318 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:33:56.578 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:33:56.579 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:34:01.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:01.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:01.581 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:34:01.619 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:34:01.619 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:34:01.887 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:34:01.887 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:34:06.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:06.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:06.888 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:34:06.929 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:34:06.929 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:34:07.192 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:34:07.193 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:34:12.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:12.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:12.194 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:34:12.234 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:34:12.234 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:34:12.475 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:34:12.476 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:34:17.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:17.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:17.477 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:34:17.520 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:34:17.521 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:34:17.792 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:34:17.792 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:34:22.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:22.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:22.793 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:34:22.835 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:34:22.835 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:34:23.111 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:34:23.111 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:34:28.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:28.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:28.113 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:34:28.155 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:34:28.155 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:34:28.436 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:34:28.437 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:34:33.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:33.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:33.437 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:34:33.478 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:34:33.478 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:34:33.762 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:34:33.762 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:34:38.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:38.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:38.763 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:34:38.803 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:34:38.803 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:34:39.061 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:34:39.061 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:34:44.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:44.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:44.062 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:34:44.103 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:34:44.103 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:34:44.379 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:34:44.380 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:34:49.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:49.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:49.381 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:34:49.422 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:34:49.422 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:34:49.692 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:34:49.693 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:34:54.722 INFO:tasks.rgw_multisite_tests:test that events are only sent on incremental sync ... FAIL 2021-02-11T02:34:54.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:54.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:54.724 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:34:54.764 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:34:54.764 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:34:54.997 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T02:34:55.041 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T02:34:55.042 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T02:34:55.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:55.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:55.042 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:34:55.082 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:34:55.083 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:34:55.328 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:34:55.328 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T02:34:55.328 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T02:34:55.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:55.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:55.329 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:34:55.372 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:34:55.372 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:34:55.630 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:34:55.630 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T02:34:55.630 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T02:34:55.630 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:55.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:34:55.631 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:34:55.670 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:34:55.670 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:34:55.930 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:34:55.930 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:35:00.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:00.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:00.931 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:35:00.972 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:35:00.972 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:35:01.233 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:35:01.234 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:35:06.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:06.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:06.236 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:35:06.277 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:35:06.277 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:35:06.541 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:35:06.541 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:35:11.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:11.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:11.542 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:35:11.585 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:35:11.585 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:35:11.842 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:35:11.842 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:35:16.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:16.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:16.843 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:35:16.885 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:35:16.885 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:35:17.183 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:35:17.183 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:35:22.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:22.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:22.184 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:35:22.225 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:35:22.225 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:35:22.498 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:35:22.498 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:35:27.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:27.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:27.499 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:35:27.539 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:35:27.539 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:35:27.805 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:35:27.805 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:35:32.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:32.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:32.806 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:35:32.849 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:35:32.849 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:35:33.120 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:35:33.121 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:35:38.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:38.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:38.487 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:35:38.527 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:35:38.527 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:35:38.823 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:35:38.824 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:35:43.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:43.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:43.825 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:35:43.866 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:35:43.867 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:35:44.144 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:35:44.144 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:35:49.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:49.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:49.146 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:35:49.188 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:35:49.188 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:35:49.459 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:35:49.460 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:35:54.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:54.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:54.461 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:35:54.503 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:35:54.504 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:35:54.787 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:35:54.787 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:35:59.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:59.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:35:59.788 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:35:59.827 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:35:59.827 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:36:00.097 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:36:00.097 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:36:05.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:05.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:05.100 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:36:05.139 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:36:05.140 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:36:05.425 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:36:05.425 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:36:10.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:10.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:10.427 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:36:10.466 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:36:10.467 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:36:10.739 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:36:10.739 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:36:15.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:15.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:15.741 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:36:15.782 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:36:15.782 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:36:16.055 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:36:16.056 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:36:21.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:21.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:21.057 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:36:21.098 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:36:21.099 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:36:21.363 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:36:21.363 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:36:26.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:26.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:26.365 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:36:26.405 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:36:26.406 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:36:26.672 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:36:26.672 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:36:31.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:31.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:31.673 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:36:31.714 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:36:31.714 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:36:31.991 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:36:31.991 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:36:36.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:36.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:36.992 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:36:37.031 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:36:37.032 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:36:37.292 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:36:37.292 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:36:42.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:42.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:42.293 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:36:42.332 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:36:42.332 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:36:42.592 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:36:42.593 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:36:47.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:47.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:47.594 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:36:47.636 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:36:47.636 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:36:47.910 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:36:47.910 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:36:52.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:52.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:52.912 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:36:52.955 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:36:52.955 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:36:53.212 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:36:53.212 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:36:58.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:58.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:36:58.214 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:36:58.261 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:36:58.261 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:36:58.516 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:36:58.517 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:37:03.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:03.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:03.518 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:37:03.558 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:37:03.558 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:37:03.819 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:37:03.820 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:37:08.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:08.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:08.821 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:37:08.862 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:37:08.862 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:37:09.136 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:37:09.136 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:37:14.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:14.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:14.137 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:37:14.178 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:37:14.178 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:37:14.453 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:37:14.454 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:37:19.455 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:19.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:19.455 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:37:19.495 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:37:19.496 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:37:19.766 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:37:19.766 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:37:24.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:24.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:24.767 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:37:24.807 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:37:26.004 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:37:26.005 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:37:26.006 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:37:31.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:31.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:31.007 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:37:31.049 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:37:31.049 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:37:31.335 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:37:31.336 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:37:36.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:36.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:36.338 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:37:36.379 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:37:36.379 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:37:36.656 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:37:36.656 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:37:41.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:41.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:41.658 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:37:41.702 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:37:41.702 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:37:41.978 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:37:41.978 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:37:46.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:46.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:46.980 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:37:47.022 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:37:47.022 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:37:47.276 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:37:47.276 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:37:52.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:52.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:52.278 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:37:52.318 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:37:52.318 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:37:52.604 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:37:52.604 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:37:57.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:57.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:37:57.606 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:37:57.648 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:37:57.648 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:37:57.931 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:37:57.931 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:38:02.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:02.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:02.933 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:38:02.974 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:38:02.974 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:38:03.256 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:38:03.256 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:38:08.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:08.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:08.259 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:38:08.300 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:38:08.300 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:38:08.576 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:38:08.576 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:38:13.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:13.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:13.577 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:38:13.620 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:38:13.620 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:38:13.900 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:38:13.900 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:38:18.902 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:18.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:18.902 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:38:18.944 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:38:18.944 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:38:19.261 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:38:19.261 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:38:24.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:24.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:24.262 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:38:24.305 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:38:24.305 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:38:24.593 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:38:24.593 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:38:29.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:29.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:29.594 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:38:29.636 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:38:29.636 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:38:29.904 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:38:29.905 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:38:34.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:34.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:34.906 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:38:34.948 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:38:34.948 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:38:35.202 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:38:35.203 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:38:40.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:40.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:40.205 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:38:40.246 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:38:40.246 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:38:40.541 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:38:40.541 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:38:45.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:45.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:45.543 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:38:45.584 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:38:45.585 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:38:45.885 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:38:45.885 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:38:50.886 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:50.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:50.887 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:38:50.928 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:38:50.928 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:38:51.197 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:38:51.198 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:38:56.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:56.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:38:56.199 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:38:56.241 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:38:56.241 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:38:56.507 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:38:56.508 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:39:01.509 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:01.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:01.509 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:39:01.552 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:39:01.552 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:39:01.813 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:39:01.813 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:39:06.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:06.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:06.814 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:39:06.855 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:39:06.855 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:39:07.128 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:39:07.128 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:39:12.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:12.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:12.129 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:39:12.172 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:39:12.172 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:39:12.436 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:39:12.437 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:39:17.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:17.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:17.438 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:39:17.480 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:39:17.481 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:39:17.750 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:39:17.750 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:39:22.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:22.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:22.752 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:39:22.792 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:39:22.792 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:39:23.059 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:39:23.059 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:39:28.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:28.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:28.061 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:39:28.103 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:39:28.103 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:39:28.384 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:39:28.384 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:39:33.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:33.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:33.386 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:39:33.426 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:39:33.426 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:39:33.682 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:39:33.682 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:39:38.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:38.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:38.684 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:39:38.723 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:39:38.724 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:39:38.993 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:39:38.994 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:39:43.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:44.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:44.734 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:39:44.776 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:39:44.776 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:39:45.050 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:39:45.050 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:39:50.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:50.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:50.052 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:39:50.091 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:39:50.091 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:39:50.384 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:39:50.384 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:39:55.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:55.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:39:55.386 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:39:55.430 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:39:55.430 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:39:55.681 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:39:55.682 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:40:00.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:00.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:00.683 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:40:00.726 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:40:00.727 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:40:01.016 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:40:01.017 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:40:06.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:06.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:06.018 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:40:06.060 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:40:06.060 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:40:06.341 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:40:06.341 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:40:11.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:11.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:11.342 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:40:11.383 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:40:11.383 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:40:11.631 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:40:11.631 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:40:16.654 INFO:tasks.rgw_multisite_tests:test subscriptions for different events ... FAIL 2021-02-11T02:40:16.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:16.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:16.655 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:40:16.694 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:40:16.694 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:40:16.934 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T02:40:16.979 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T02:40:16.979 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T02:40:16.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:16.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:16.980 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:40:17.021 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:40:17.021 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:40:17.283 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:40:17.284 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T02:40:17.284 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T02:40:17.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:17.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:17.284 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:40:17.324 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:40:17.324 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:40:17.614 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:40:17.614 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T02:40:17.615 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T02:40:17.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:17.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:17.615 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:40:17.655 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:40:17.655 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:40:17.942 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:40:17.943 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:40:22.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:22.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:22.944 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:40:22.985 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:40:22.985 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:40:23.252 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:40:23.253 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:40:28.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:28.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:28.254 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:40:28.292 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:40:28.292 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:40:28.558 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:40:28.558 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:40:33.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:33.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:33.560 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:40:33.601 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:40:33.602 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:40:33.889 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:40:33.889 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:40:38.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:38.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:38.891 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:40:38.932 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:40:38.932 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:40:39.225 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:40:39.225 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:40:44.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:44.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:44.227 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:40:44.271 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:40:44.272 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:40:44.523 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:40:44.524 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:40:49.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:49.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:49.526 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:40:49.567 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:40:49.567 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:40:49.824 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:40:49.825 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:40:54.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:55.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:40:55.679 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:40:55.719 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:40:55.719 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:40:55.992 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:40:55.992 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:41:00.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:00.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:00.993 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:41:01.034 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:41:01.035 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:41:01.303 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:41:01.303 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:41:06.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:06.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:06.305 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:41:06.348 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:41:06.348 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:41:06.611 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:41:06.612 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:41:11.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:11.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:11.613 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:41:11.655 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:41:11.655 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:41:11.934 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:41:11.934 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:41:16.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:16.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:16.935 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:41:16.975 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:41:16.975 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:41:17.259 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:41:17.259 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:41:22.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:22.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:22.260 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:41:22.299 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:41:22.300 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:41:22.595 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:41:22.596 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:41:27.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:27.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:27.597 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:41:27.638 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:41:27.639 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:41:27.915 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:41:27.915 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:41:32.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:32.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:32.917 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:41:32.957 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:41:32.958 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:41:33.236 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:41:33.236 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:41:38.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:38.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:38.238 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:41:38.279 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:41:38.280 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:41:38.571 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:41:38.571 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:41:43.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:43.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:43.573 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:41:43.614 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:41:43.614 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:41:43.890 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:41:43.890 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:41:48.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:48.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:48.892 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:41:48.933 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:41:48.933 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:41:49.187 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:41:49.188 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:41:54.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:54.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:54.189 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:41:54.231 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:41:54.232 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:41:54.506 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:41:54.507 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:41:59.507 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:59.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:41:59.508 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:41:59.549 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:41:59.549 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:41:59.808 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:41:59.808 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:42:04.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:04.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:04.810 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:42:04.852 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:42:04.852 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:42:05.115 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:42:05.116 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:42:10.117 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:10.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:10.117 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:42:10.160 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:42:10.160 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:42:10.424 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:42:10.425 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:42:15.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:15.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:15.426 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:42:15.466 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:42:15.467 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:42:15.725 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:42:15.725 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:42:20.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:20.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:20.727 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:42:20.770 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:42:20.771 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:42:21.030 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:42:21.030 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:42:26.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:26.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:26.032 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:42:26.073 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:42:26.073 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:42:26.340 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:42:26.340 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:42:31.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:31.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:31.341 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:42:31.384 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:42:31.384 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:42:31.660 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:42:31.661 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:42:36.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:36.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:36.662 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:42:36.703 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:42:36.703 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:42:36.972 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:42:36.972 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:42:41.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:41.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:41.974 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:42:42.016 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:42:42.016 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:42:42.294 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:42:42.295 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:42:47.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:47.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:47.296 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:42:47.333 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:42:47.333 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:42:47.613 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:42:47.613 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:42:52.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:52.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:52.614 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:42:52.657 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:42:52.657 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:42:52.927 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:42:52.927 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:42:57.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:57.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:42:57.929 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:42:57.970 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:42:57.970 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:42:58.244 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:42:58.244 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:43:03.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:03.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:03.246 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:43:03.290 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:43:03.290 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:43:03.556 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:43:03.556 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:43:08.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:08.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:08.558 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:43:08.598 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:43:08.599 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:43:08.856 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:43:08.856 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:43:13.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:15.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:15.191 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:43:15.231 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:43:15.231 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:43:15.495 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:43:15.496 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:43:20.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:20.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:20.497 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:43:20.538 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:43:20.538 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:43:20.805 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:43:20.806 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:43:25.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:25.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:25.807 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:43:25.849 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:43:25.849 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:43:26.110 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:43:26.111 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:43:31.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:31.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:31.112 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:43:31.152 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:43:31.153 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:43:31.431 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:43:31.432 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:43:36.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:36.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:36.433 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:43:36.473 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:43:36.473 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:43:36.734 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:43:36.735 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:43:41.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:41.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:41.737 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:43:41.779 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:43:41.779 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:43:42.049 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:43:42.050 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:43:47.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:47.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:47.051 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:43:47.092 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:43:47.092 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:43:47.349 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:43:47.350 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:43:52.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:52.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:52.352 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:43:52.390 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:43:52.391 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:43:52.652 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:43:52.652 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:43:57.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:57.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:43:57.653 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:43:57.694 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:43:57.694 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:43:57.964 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:43:57.964 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:44:02.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:02.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:02.966 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:44:03.010 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:44:03.010 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:44:03.285 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:44:03.286 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:44:08.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:08.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:08.287 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:44:08.319 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:44:08.319 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:44:08.583 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:44:08.584 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:44:13.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:13.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:13.585 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:44:13.628 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:44:13.628 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:44:13.918 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:44:13.919 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:44:18.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:18.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:18.921 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:44:18.963 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:44:18.963 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:44:19.219 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:44:19.220 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:44:24.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:24.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:24.396 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:44:24.437 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:44:24.438 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:44:24.708 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:44:25.624 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:44:30.625 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:30.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:30.626 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:44:30.668 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:44:30.668 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:44:30.945 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:44:30.945 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:44:35.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:35.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:35.946 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:44:35.987 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:44:35.987 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:44:36.276 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:44:36.278 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:44:41.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:41.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:41.280 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:44:41.320 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:44:41.320 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:44:41.583 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:44:41.583 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:44:46.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:46.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:46.584 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:44:46.640 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:44:46.641 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:44:46.917 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:44:46.917 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:44:51.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:51.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:51.919 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:44:51.960 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:44:51.960 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:44:52.238 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:44:52.239 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:44:57.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:57.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:44:57.240 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:44:57.279 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:44:57.279 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:44:57.538 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:44:57.539 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:45:02.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:02.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:02.540 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:45:02.579 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:45:02.580 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:45:02.850 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:45:02.850 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:45:07.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:09.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:09.009 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:45:09.052 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:45:09.052 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:45:09.330 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:45:09.330 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:45:14.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:14.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:14.332 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:45:14.373 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:45:14.374 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:45:14.648 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:45:14.648 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:45:19.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:19.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:19.650 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:45:19.691 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:45:19.692 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:45:19.957 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:45:19.958 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:45:24.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:24.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:24.959 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:45:25.001 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:45:25.001 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:45:25.293 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:45:25.293 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:45:30.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:30.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:30.294 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:45:30.332 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:45:30.332 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:45:30.599 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:45:30.600 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:45:35.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:35.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:35.601 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:45:35.640 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:45:35.640 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:45:35.908 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:45:35.908 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:45:40.933 INFO:tasks.rgw_multisite_tests:test incremental fetching of events from a subscription ... FAIL 2021-02-11T02:45:42.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:42.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:42.888 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:45:42.927 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:45:42.927 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:45:43.171 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T02:45:43.211 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T02:45:43.212 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T02:45:43.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:43.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:43.212 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:45:43.253 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:45:43.253 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:45:43.509 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:45:43.510 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T02:45:43.510 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T02:45:43.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:43.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:43.510 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:45:43.552 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:45:43.552 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:45:43.842 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:45:43.842 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T02:45:43.843 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T02:45:43.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:43.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:43.843 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:45:43.883 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:45:43.883 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:45:44.154 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:45:44.155 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:45:49.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:49.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:49.807 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:45:49.847 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:45:49.848 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:45:50.122 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:45:51.774 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:45:56.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:56.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:45:56.775 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:45:56.816 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:45:56.817 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:45:57.075 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:45:57.076 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:46:02.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:02.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:02.078 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:46:02.120 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:46:02.120 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:46:02.374 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:46:02.374 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:46:07.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:07.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:07.375 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:46:07.416 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:46:07.417 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:46:07.691 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:46:07.691 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:46:12.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:14.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:14.208 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:46:14.249 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:46:14.249 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:46:14.513 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:46:14.513 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:46:19.514 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:19.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:19.515 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:46:19.554 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:46:19.554 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:46:19.827 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:46:19.828 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:46:24.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:24.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:24.829 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:46:24.868 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:46:24.868 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:46:25.144 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:46:25.145 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:46:30.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:30.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:30.146 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:46:30.188 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:46:30.188 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:46:30.456 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:46:30.456 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:46:35.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:35.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:35.458 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:46:35.497 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:46:35.497 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:46:35.767 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:46:35.768 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:46:40.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:40.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:40.769 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:46:40.810 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:46:40.810 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:46:41.082 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:46:41.083 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:46:46.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:46.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:46.211 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:46:46.251 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:46:46.251 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:46:46.495 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:46:46.496 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:46:51.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:51.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:51.497 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:46:51.536 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:46:51.536 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:46:51.800 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:46:51.800 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:46:56.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:56.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:46:56.802 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:46:56.842 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:46:56.842 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:46:57.121 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:46:57.121 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:47:02.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:02.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:02.123 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:47:02.165 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:47:02.165 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:47:02.427 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:47:02.428 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:47:07.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:07.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:07.429 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:47:07.467 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:47:07.467 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:47:07.733 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:47:07.733 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:47:12.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:12.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:12.734 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:47:12.776 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:47:12.777 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:47:13.043 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:47:13.043 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:47:18.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:18.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:18.045 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:47:18.086 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:47:18.086 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:47:18.363 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:47:18.363 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:47:23.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:23.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:23.364 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:47:23.403 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:47:23.403 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:47:23.692 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:47:23.693 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:47:28.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:28.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:28.694 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:47:28.736 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:47:28.736 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:47:29.006 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:47:29.006 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:47:34.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:34.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:34.007 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:47:34.047 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:47:34.047 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:47:34.314 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:47:34.314 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:47:39.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:39.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:39.316 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:47:39.357 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:47:39.357 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:47:39.623 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:47:39.624 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:47:44.625 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:44.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:44.626 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:47:44.668 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:47:44.668 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:47:44.951 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:47:44.952 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:47:49.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:49.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:49.954 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:47:49.995 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:47:49.995 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:47:50.280 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:47:50.280 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:47:55.281 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:55.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:47:55.281 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:47:55.317 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:47:55.318 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:47:55.572 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:47:55.573 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:48:00.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:00.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:00.574 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:48:00.617 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:48:00.617 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:48:00.879 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:48:00.880 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:48:05.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:05.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:05.881 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:48:05.920 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:48:05.920 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:48:06.192 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:48:06.192 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:48:11.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:11.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:11.194 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:48:11.241 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:48:11.241 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:48:11.485 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:48:11.485 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:48:16.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:16.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:16.487 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:48:16.528 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:48:16.528 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:48:16.782 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:48:16.782 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:48:21.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:21.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:21.951 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:48:21.991 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:48:21.991 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:48:22.281 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:48:22.282 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:48:27.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:27.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:27.283 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:48:27.320 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:48:27.320 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:48:27.590 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:48:27.590 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:48:32.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:32.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:32.592 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:48:32.633 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:48:32.633 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:48:32.901 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:48:32.902 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:48:37.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:37.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:37.903 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:48:37.943 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:48:37.943 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:48:38.221 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:48:38.222 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:48:43.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:43.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:43.223 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:48:43.270 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:48:43.270 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:48:43.532 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:48:43.533 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:48:48.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:48.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:48.534 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:48:48.575 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:48:48.575 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:48:48.854 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:48:48.855 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:48:53.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:53.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:53.856 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:48:53.897 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:48:53.897 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:48:54.154 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:48:54.154 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:48:59.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:59.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:48:59.156 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:48:59.195 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:48:59.195 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:48:59.456 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:48:59.456 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:49:04.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:04.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:04.458 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:49:04.502 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:49:04.502 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:49:04.771 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:49:04.772 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:49:09.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:09.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:09.773 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:49:09.814 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:49:09.814 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:49:10.109 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:49:10.109 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:49:15.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:15.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:15.110 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:49:15.153 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:49:15.153 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:49:15.426 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:49:15.426 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:49:20.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:20.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:20.427 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:49:20.468 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:49:20.468 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:49:20.754 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:49:21.490 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:49:26.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:26.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:26.492 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:49:26.531 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:49:26.531 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:49:26.796 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:49:26.797 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:49:31.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:31.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:31.798 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:49:31.840 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:49:31.841 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:49:32.129 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:49:32.130 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:49:37.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:37.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:37.131 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:49:37.173 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:49:37.174 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:49:37.421 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:49:37.421 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:49:42.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:42.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:42.423 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:49:42.466 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:49:42.466 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:49:42.732 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:49:42.733 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:49:47.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:47.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:47.734 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:49:47.775 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:49:47.775 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:49:48.045 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:49:48.045 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:49:53.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:53.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:53.047 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:49:53.089 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:49:53.089 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:49:53.338 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:49:53.794 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:49:58.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:58.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:49:58.795 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:49:58.836 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:49:58.836 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:49:59.089 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:49:59.089 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:50:04.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:04.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:04.091 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:50:04.135 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:50:04.135 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:50:04.387 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:50:04.388 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:50:09.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:09.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:09.390 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:50:09.432 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:50:09.432 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:50:09.694 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:50:09.695 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:50:14.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:14.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:14.696 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:50:14.736 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:50:14.736 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:50:14.992 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:50:14.992 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:50:19.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:19.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:19.993 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:50:20.034 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:50:20.035 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:50:20.309 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:50:20.310 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:50:25.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:25.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:25.311 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:50:25.347 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:50:25.347 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:50:25.620 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:50:25.620 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:50:30.621 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:30.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:30.621 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:50:30.662 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:50:30.663 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:50:30.931 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:50:30.931 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:50:35.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:35.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:35.933 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:50:35.974 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:50:35.974 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:50:36.240 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:50:36.241 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:50:41.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:41.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:41.242 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:50:41.284 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:50:41.284 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:50:41.548 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:50:41.548 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:50:46.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:46.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:46.549 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:50:46.591 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:50:46.591 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:50:46.843 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:50:46.843 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:50:51.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:51.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:51.845 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:50:51.886 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:50:51.886 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:50:52.155 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:50:52.155 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:50:57.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:57.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:50:57.157 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:50:57.200 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:50:57.200 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:50:57.464 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:50:57.464 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:51:02.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:02.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:02.466 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:51:02.505 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:51:02.505 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:51:02.768 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:51:03.694 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:51:08.717 INFO:tasks.rgw_multisite_tests:test acking of some events in a subscription ... FAIL 2021-02-11T02:51:08.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:08.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:08.719 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:51:08.760 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:51:08.760 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:51:09.014 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T02:51:09.058 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T02:51:09.058 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T02:51:09.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:09.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:09.059 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:51:09.101 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:51:09.101 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:51:09.357 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:51:09.357 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T02:51:09.357 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T02:51:09.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:09.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:09.359 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:51:09.400 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:51:09.400 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:51:09.670 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:51:09.670 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T02:51:09.671 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T02:51:09.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:09.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:09.671 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:51:09.711 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:51:09.711 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:51:09.968 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:51:09.968 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:51:14.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:17.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:17.332 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:51:17.372 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:51:17.372 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:51:17.649 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:51:17.649 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:51:22.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:22.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:22.651 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:51:22.693 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:51:22.693 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:51:22.975 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:51:22.975 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:51:27.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:27.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:27.976 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:51:28.016 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:51:28.016 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:51:28.300 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:51:28.301 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:51:33.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:33.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:33.302 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:51:33.343 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:51:33.343 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:51:33.614 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:51:33.614 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:51:38.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:38.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:38.615 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:51:38.655 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:51:38.655 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:51:38.932 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:51:38.932 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:51:43.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:43.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:43.934 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:51:43.974 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:51:43.975 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:51:44.252 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:51:44.252 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:51:49.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:49.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:49.254 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:51:49.297 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:51:49.297 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:51:49.565 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:51:51.221 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:51:56.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:56.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:51:56.222 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:51:56.264 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:51:56.265 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:51:56.521 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:51:56.521 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:52:01.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:01.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:01.523 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:52:01.567 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:52:01.567 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:52:01.845 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:52:01.845 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:52:06.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:06.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:06.847 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:52:06.888 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:52:06.888 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:52:07.162 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:52:07.162 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:52:12.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:12.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:12.164 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:52:12.205 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:52:12.205 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:52:12.492 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:52:12.492 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:52:17.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:17.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:17.493 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:52:17.536 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:52:17.536 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:52:17.800 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:52:17.801 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:52:22.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:22.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:22.802 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:52:22.843 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:52:22.843 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:52:23.137 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:52:23.137 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:52:28.139 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:28.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:28.139 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:52:28.181 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:52:28.181 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:52:28.454 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:52:28.455 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:52:33.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:33.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:33.457 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:52:33.499 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:52:33.499 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:52:33.766 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:52:33.767 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:52:38.767 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:38.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:38.768 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:52:38.808 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:52:38.809 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:52:39.093 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:52:39.093 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:52:44.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:44.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:44.095 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:52:44.137 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:52:44.137 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:52:44.422 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:52:44.423 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:52:49.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:49.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:49.424 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:52:49.465 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:52:49.466 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:52:49.733 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:52:49.733 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:52:54.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:54.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:52:54.735 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:52:54.777 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:52:54.777 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:52:55.037 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:52:55.037 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:53:00.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:00.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:00.039 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:53:00.078 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:53:00.078 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:53:00.330 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:53:00.330 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:53:05.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:05.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:05.999 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:53:06.042 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:53:06.043 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:53:06.310 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:53:06.311 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:53:11.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:11.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:11.313 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:53:11.349 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:53:11.349 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:53:11.611 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:53:11.611 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:53:16.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:19.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:19.461 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:53:19.504 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:53:19.504 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:53:19.788 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:53:21.213 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:53:26.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:26.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:26.215 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:53:26.257 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:53:26.257 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:53:26.559 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:53:26.559 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:53:31.560 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:31.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:31.561 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:53:31.604 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:53:31.604 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:53:31.859 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:53:31.859 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:53:36.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:36.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:36.860 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:53:36.900 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:53:36.900 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:53:37.170 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:53:40.038 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:53:45.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:45.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:45.039 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:53:45.083 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:53:45.083 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:53:45.363 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:53:45.363 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:53:50.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:50.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:50.365 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:53:50.400 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:53:50.400 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:53:50.686 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:53:52.277 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:53:57.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:57.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:53:57.279 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:53:57.322 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:53:57.322 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:53:57.584 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:53:57.584 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:54:02.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:02.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:02.586 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:54:02.631 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:54:02.632 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:54:02.903 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:54:02.904 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:54:07.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:07.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:07.905 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:54:07.946 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:54:07.946 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:54:08.228 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:54:08.228 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:54:13.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:13.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:13.230 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:54:13.279 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:54:13.279 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:54:13.543 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:54:13.543 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:54:18.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:18.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:18.545 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:54:18.584 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:54:18.585 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:54:18.854 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:54:18.854 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:54:23.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:23.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:23.856 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:54:23.897 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:54:23.897 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:54:24.162 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:54:24.162 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:54:29.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:29.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:29.163 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:54:29.205 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:54:29.205 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:54:29.464 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:54:29.464 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:54:34.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:34.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:34.465 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:54:34.504 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:54:34.504 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:54:34.773 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:54:34.773 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:54:39.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:39.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:39.775 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:54:39.815 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:54:39.815 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:54:40.086 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:54:40.086 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:54:45.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:45.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:45.088 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:54:45.130 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:54:45.130 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:54:45.410 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:54:45.410 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:54:50.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:50.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:50.412 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:54:50.453 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:54:50.454 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:54:50.729 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:54:50.729 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:54:55.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:55.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:54:55.731 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:54:55.774 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:54:55.775 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:54:56.050 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:54:56.051 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:55:01.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:01.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:01.052 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:55:01.093 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:55:01.094 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:55:01.370 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:55:01.371 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:55:06.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:06.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:06.373 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:55:06.408 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:55:06.408 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:55:06.676 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:55:06.676 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:55:11.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:11.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:11.678 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:55:11.720 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:55:11.720 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:55:12.010 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:55:12.868 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:55:17.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:17.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:17.870 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:55:17.911 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:55:17.911 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:55:18.173 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:55:18.173 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:55:23.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:23.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:23.174 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:55:23.215 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:55:23.215 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:55:23.490 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:55:23.491 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:55:28.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:28.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:28.492 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:55:28.531 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:55:28.531 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:55:28.793 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:55:28.794 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:55:33.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:33.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:33.796 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:55:33.837 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:55:33.837 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:55:34.102 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:55:34.102 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:55:39.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:39.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:39.103 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:55:39.143 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:55:40.737 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:55:40.741 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:55:40.742 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:55:45.744 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:45.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:45.745 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:55:45.784 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:55:45.784 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:55:46.057 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:55:46.057 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:55:51.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:51.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:51.058 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:55:51.115 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:55:51.115 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:55:51.356 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:55:51.357 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:55:56.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:56.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:55:56.359 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:55:56.398 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:55:56.398 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:55:56.673 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:55:56.673 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:56:01.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:01.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:01.674 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:56:01.719 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:56:01.719 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:56:01.989 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:56:01.989 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:56:06.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:06.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:06.991 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:56:07.033 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:56:07.033 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:56:07.289 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:56:07.290 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:56:12.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:13.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:13.298 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:56:13.340 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:56:13.340 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:56:13.597 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:56:13.597 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:56:18.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:18.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:18.599 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:56:18.641 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:56:18.641 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:56:18.908 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:56:18.909 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:56:23.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:23.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:23.910 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:56:23.950 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:56:23.950 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:56:24.225 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:56:24.226 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:56:29.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:29.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:29.227 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:56:29.272 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:56:29.272 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:56:29.529 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:56:29.529 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:56:34.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:34.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:34.530 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:56:34.571 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:56:34.571 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:56:34.872 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:56:34.872 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:56:39.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:39.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:39.874 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:56:39.916 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:56:39.917 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:56:40.179 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:56:40.179 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:56:45.202 INFO:tasks.rgw_multisite_tests:test object creation notifications in using put/copy/post ... FAIL 2021-02-11T02:56:45.203 INFO:tasks.rgw_multisite_tests:test object creation s3 notifications in using put/copy/post on master ... ok 2021-02-11T02:56:45.203 INFO:tasks.rgw_multisite_tests:test multipart object upload on master ... ok 2021-02-11T02:56:45.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:45.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:45.204 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:56:45.245 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:56:45.245 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:56:45.489 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T02:56:45.533 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T02:56:45.534 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T02:56:45.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:45.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:45.534 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:56:45.575 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:56:45.575 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:56:45.840 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:56:45.840 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T02:56:45.840 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T02:56:45.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:45.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:45.841 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:56:45.882 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:56:45.883 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:56:46.163 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:56:46.163 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T02:56:46.163 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T02:56:46.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:46.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:46.164 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:56:46.203 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:56:46.203 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:56:46.449 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:56:46.450 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:56:51.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:51.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:51.451 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:56:51.493 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:56:51.493 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:56:51.756 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:56:51.756 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:56:56.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:56.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:56:56.759 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:56:56.796 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:56:56.797 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:56:57.076 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:56:57.076 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:57:02.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:02.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:02.078 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:57:02.118 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:57:02.118 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:57:02.375 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:57:02.375 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:57:07.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:07.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:07.377 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:57:07.418 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:57:07.418 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:57:07.680 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:57:07.680 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:57:12.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:12.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:12.681 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:57:12.722 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:57:12.722 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:57:13.006 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:57:13.007 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:57:18.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:18.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:18.008 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:57:18.049 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:57:18.049 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:57:18.324 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:57:18.324 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:57:23.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:23.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:23.326 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:57:23.369 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:57:23.369 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:57:23.629 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:57:23.629 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:57:28.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:28.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:28.631 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:57:28.673 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:57:28.673 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:57:28.937 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:57:28.938 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:57:33.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:33.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:33.939 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:57:33.981 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:57:33.981 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:57:34.269 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:57:34.269 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:57:39.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:39.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:39.271 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:57:39.313 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:57:39.313 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:57:39.592 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:57:39.592 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:57:44.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:44.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:44.593 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:57:44.635 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:57:44.635 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:57:44.895 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:57:44.895 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:57:49.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:49.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:49.897 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:57:49.936 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:57:49.936 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:57:50.210 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:57:50.211 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:57:55.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:55.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:57:55.212 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:57:55.253 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:57:55.253 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:57:55.545 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:57:55.546 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:58:00.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:00.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:00.548 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:58:00.589 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:58:00.589 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:58:00.841 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:58:00.841 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:58:05.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:05.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:05.843 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:58:05.884 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:58:05.884 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:58:06.147 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:58:06.147 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:58:11.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:11.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:11.580 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:58:11.621 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:58:11.621 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:58:11.896 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:58:11.896 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:58:16.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:16.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:16.897 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:58:16.940 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:58:16.940 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:58:17.202 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:58:17.203 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:58:22.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:22.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:22.205 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:58:22.244 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:58:22.244 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:58:22.515 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:58:22.515 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:58:27.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:27.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:27.517 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:58:27.558 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:58:27.558 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:58:27.831 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:58:27.831 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:58:32.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:32.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:32.832 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:58:32.872 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:58:32.872 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:58:33.130 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:58:33.130 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:58:38.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:38.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:38.131 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:58:38.172 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:58:38.172 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:58:38.428 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:58:38.428 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:58:43.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:43.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:43.430 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:58:43.471 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:58:43.471 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:58:43.761 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:58:43.762 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:58:48.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:48.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:48.763 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:58:48.805 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:58:48.805 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:58:49.088 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:58:49.089 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:58:54.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:54.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:54.090 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:58:54.130 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:58:54.131 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:58:54.384 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:58:54.385 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:58:59.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:59.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:58:59.386 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:58:59.428 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:58:59.428 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:58:59.705 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:58:59.705 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:59:04.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:04.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:04.707 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:59:04.749 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:59:04.749 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:59:05.013 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:59:05.014 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:59:10.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:10.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:10.016 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:59:10.057 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:59:10.057 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:59:10.333 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:59:10.333 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:59:15.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:15.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:15.334 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:59:15.376 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:59:15.377 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:59:15.649 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:59:15.650 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:59:20.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:20.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:20.652 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:59:20.694 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:59:20.694 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:59:20.956 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:59:22.103 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:59:27.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:27.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:27.105 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:59:27.146 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:59:27.146 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:59:27.406 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:59:27.406 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:59:32.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:32.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:32.408 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:59:32.451 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:59:32.451 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:59:32.720 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:59:32.721 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:59:37.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:37.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:37.722 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:59:37.764 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:59:37.764 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:59:38.037 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:59:38.037 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:59:43.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:43.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:43.819 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:59:43.863 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:59:43.863 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:59:44.137 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:59:44.138 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:59:49.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:49.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:49.139 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:59:49.179 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:59:49.179 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:59:49.450 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:59:49.450 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:59:54.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:54.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:54.452 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:59:54.491 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:59:54.491 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T02:59:54.766 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T02:59:54.767 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T02:59:59.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:59.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T02:59:59.768 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T02:59:59.809 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T02:59:59.809 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:00:00.084 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:00:00.084 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:00:05.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:05.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:05.086 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:00:05.130 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:00:05.130 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:00:05.401 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:00:05.401 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:00:10.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:10.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:10.403 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:00:10.445 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:00:10.446 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:00:10.711 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:00:10.711 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:00:15.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:15.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:15.713 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:00:15.754 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:00:15.754 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:00:16.019 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:00:16.019 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:00:21.020 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:21.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:21.020 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:00:21.062 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:00:21.063 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:00:21.337 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:00:21.337 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:00:26.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:26.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:26.338 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:00:26.378 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:00:26.378 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:00:26.652 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:00:26.653 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:00:31.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:31.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:31.654 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:00:31.696 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:00:31.696 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:00:31.961 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:00:31.961 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:00:36.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:36.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:36.963 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:00:37.003 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:00:37.003 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:00:37.278 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:00:37.278 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:00:42.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:42.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:42.280 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:00:42.320 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:00:42.320 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:00:42.578 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:00:42.579 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:00:47.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:47.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:47.580 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:00:47.620 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:00:47.621 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:00:47.891 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:00:47.892 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:00:52.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:52.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:52.894 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:00:52.936 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:00:52.936 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:00:53.215 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:00:53.216 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:00:58.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:58.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:00:58.216 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:00:58.256 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:00:58.257 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:00:58.512 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:00:58.513 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:01:03.514 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:03.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:03.514 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:01:03.556 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:01:03.556 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:01:03.840 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:01:03.841 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:01:08.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:08.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:08.842 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:01:08.884 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:01:08.884 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:01:09.162 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:01:09.162 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:01:14.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:14.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:14.164 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:01:14.204 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:01:14.204 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:01:14.455 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:01:14.455 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:01:19.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:19.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:19.457 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:01:19.498 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:01:19.498 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:01:19.768 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:01:19.769 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:01:24.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:24.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:24.770 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:01:24.812 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:01:24.813 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:01:25.090 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:01:25.091 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:01:30.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:30.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:30.092 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:01:30.133 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:01:30.134 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:01:30.408 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:01:30.408 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:01:35.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:35.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:35.410 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:01:35.449 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:01:35.449 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:01:35.702 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:01:35.702 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:01:40.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:40.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:40.704 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:01:40.745 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:01:40.745 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:01:41.014 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:01:41.015 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:01:46.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:46.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:46.016 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:01:46.056 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:01:46.056 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:01:46.327 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:01:46.328 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:01:51.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:51.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:51.330 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:01:51.368 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:01:51.368 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:01:51.636 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:01:51.636 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:01:56.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:56.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:01:56.638 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:01:56.678 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:01:56.678 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:01:56.951 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:01:56.951 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:02:01.952 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:01.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:01.952 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:02:01.992 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:02:01.992 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:02:02.261 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:02:02.261 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:02:07.285 INFO:tasks.rgw_multisite_tests:test notification of deletion markers ... FAIL 2021-02-11T03:02:07.285 INFO:tasks.rgw_multisite_tests:test s3 notification of metadata on master ... ok 2021-02-11T03:02:07.286 INFO:tasks.rgw_multisite_tests:test s3 notification of tags on master ... ok 2021-02-11T03:02:07.286 INFO:tasks.rgw_multisite_tests:test s3 notification of object versions ... ok 2021-02-11T03:02:07.286 INFO:tasks.rgw_multisite_tests:test s3 notification of deletion markers on master ... ok 2021-02-11T03:02:07.286 INFO:tasks.rgw_multisite_tests:test pushing to http endpoint ... ok 2021-02-11T03:02:07.286 INFO:tasks.rgw_multisite_tests:test pushing to http endpoint s3 record format ... ok 2021-02-11T03:02:07.287 INFO:tasks.rgw_multisite_tests:test pushing to amqp endpoint ... ok 2021-02-11T03:02:07.287 INFO:tasks.rgw_multisite_tests:test pushing to amqp endpoint s3 record format ... ok 2021-02-11T03:02:07.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:07.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:07.287 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:02:07.328 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:02:07.329 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:02:07.581 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T03:02:08.357 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T03:02:08.357 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T03:02:08.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:08.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:08.358 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:02:08.400 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:02:08.400 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:02:08.661 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:02:08.662 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T03:02:08.662 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T03:02:08.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:08.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:08.663 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:02:08.705 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:02:08.705 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:02:08.996 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:02:08.997 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T03:02:08.997 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T03:02:08.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:08.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:08.997 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:02:09.039 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:02:09.039 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:02:09.307 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:02:09.307 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:02:14.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:14.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:14.308 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:02:14.349 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:02:14.349 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:02:14.608 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:02:14.609 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:02:19.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:19.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:19.610 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:02:19.655 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:02:19.655 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:02:19.936 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:02:19.936 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:02:24.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:24.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:24.938 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:02:24.980 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:02:24.980 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:02:25.256 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:02:25.257 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:02:30.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:30.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:30.258 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:02:30.307 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:02:30.308 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:02:30.596 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:02:30.596 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:02:35.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:35.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:35.598 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:02:35.640 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:02:35.640 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:02:35.905 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:02:35.906 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:02:40.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:40.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:40.907 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:02:40.947 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:02:40.948 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:02:41.229 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:02:41.230 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:02:46.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:46.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:46.231 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:02:46.273 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:02:46.273 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:02:46.521 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:02:46.521 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:02:51.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:51.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:51.779 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:02:51.821 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:02:51.821 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:02:52.082 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:02:52.083 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:02:57.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:57.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:02:57.084 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:02:57.124 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:02:57.124 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:02:57.375 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:02:57.375 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:03:02.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:02.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:02.377 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:03:02.421 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:03:02.421 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:03:02.681 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:03:02.681 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:03:07.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:07.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:07.682 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:03:07.724 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:03:07.724 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:03:07.991 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:03:07.991 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:03:12.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:13.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:13.771 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:03:13.812 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:03:13.812 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:03:14.077 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:03:14.078 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:03:19.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:19.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:19.079 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:03:19.121 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:03:19.121 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:03:19.411 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:03:19.412 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:03:24.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:24.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:24.414 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:03:24.455 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:03:24.456 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:03:24.721 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:03:24.722 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:03:29.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:29.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:29.723 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:03:29.765 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:03:29.765 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:03:30.029 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:03:30.029 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:03:35.029 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:35.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:35.030 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:03:35.073 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:03:35.073 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:03:35.326 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:03:35.327 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:03:40.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:40.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:40.328 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:03:40.368 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:03:40.369 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:03:40.645 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:03:40.645 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:03:45.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:45.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:45.646 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:03:45.687 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:03:45.688 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:03:45.973 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:03:45.974 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:03:50.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:50.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:50.975 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:03:51.015 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:03:51.016 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:03:51.292 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:03:51.293 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:03:56.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:56.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:03:56.295 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:03:56.336 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:03:56.337 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:03:56.617 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:03:56.618 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:04:01.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:01.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:01.620 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:04:01.659 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:04:01.659 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:04:01.937 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:04:01.938 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:04:06.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:06.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:06.939 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:04:06.981 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:04:06.981 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:04:07.234 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:04:07.234 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:04:12.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:12.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:12.236 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:04:12.277 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:04:12.277 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:04:12.551 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:04:12.552 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:04:17.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:17.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:17.553 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:04:17.594 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:04:17.594 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:04:17.863 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:04:17.864 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:04:22.865 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:23.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:23.817 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:04:23.856 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:04:23.857 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:04:24.143 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:04:24.144 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:04:29.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:29.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:29.146 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:04:29.187 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:04:29.187 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:04:29.478 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:04:29.479 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:04:34.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:34.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:34.480 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:04:34.521 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:04:34.521 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:04:34.779 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:04:34.780 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:04:39.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:39.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:39.781 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:04:39.823 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:04:39.824 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:04:40.089 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:04:40.089 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:04:45.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:45.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:45.091 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:04:45.132 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:04:45.133 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:04:45.389 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:04:45.390 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:04:50.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:50.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:50.391 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:04:50.433 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:04:50.433 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:04:50.701 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:04:50.701 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:04:55.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:55.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:04:55.703 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:04:55.746 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:04:55.746 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:04:56.018 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:04:56.018 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:05:01.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:01.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:01.020 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:05:01.062 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:05:01.062 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:05:01.337 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:05:01.338 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:05:06.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:06.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:06.339 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:05:06.381 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:05:06.381 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:05:06.648 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:05:06.648 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:05:11.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:11.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:11.649 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:05:11.691 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:05:11.691 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:05:11.970 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:05:11.970 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:05:16.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:16.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:16.972 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:05:17.013 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:05:17.013 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:05:17.278 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:05:17.278 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:05:22.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:22.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:22.280 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:05:22.320 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:05:22.320 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:05:22.588 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:05:22.589 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:05:27.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:27.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:27.590 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:05:27.633 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:05:28.500 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:05:28.501 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:05:28.502 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:05:33.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:33.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:33.503 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:05:33.543 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:05:33.544 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:05:33.857 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:05:33.858 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:05:38.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:38.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:38.859 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:05:38.900 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:05:38.900 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:05:39.161 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:05:39.161 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:05:44.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:44.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:44.163 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:05:44.203 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:05:44.204 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:05:44.467 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:05:44.467 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:05:49.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:49.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:49.469 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:05:49.508 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:05:49.509 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:05:49.776 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:05:49.777 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:05:54.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:54.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:05:54.778 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:05:54.819 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:05:54.819 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:05:55.093 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:05:56.268 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:06:01.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:01.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:01.269 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:06:01.307 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:06:01.308 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:06:01.575 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:06:01.575 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:06:06.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:06.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:06.576 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:06:06.616 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:06:06.616 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:06:06.895 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:06:06.895 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:06:11.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:11.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:11.896 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:06:11.937 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:06:11.938 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:06:12.207 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:06:12.207 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:06:17.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:17.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:17.209 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:06:17.255 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:06:17.255 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:06:17.491 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:06:17.491 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:06:22.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:22.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:22.492 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:06:22.531 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:06:22.532 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:06:22.800 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:06:22.800 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:06:27.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:27.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:27.802 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:06:27.844 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:06:27.845 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:06:28.118 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:06:28.118 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:06:33.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:33.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:33.119 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:06:33.160 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:06:33.160 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:06:33.438 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:06:34.242 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:06:39.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:39.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:39.244 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:06:39.288 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:06:39.288 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:06:39.541 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:06:39.542 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:06:44.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:44.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:44.543 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:06:44.584 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:06:44.584 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:06:44.851 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:06:44.852 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:06:49.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:49.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:49.853 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:06:49.892 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:06:49.893 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:06:50.159 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:06:50.159 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:06:55.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:55.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:06:55.161 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:06:55.200 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:06:55.200 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:06:55.456 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:06:55.456 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:07:00.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:00.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:00.458 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:07:00.491 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:07:00.491 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:07:00.762 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:07:01.822 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:07:06.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:06.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:06.823 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:07:06.863 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:07:06.863 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:07:07.147 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:07:07.147 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:07:12.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:12.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:12.148 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:07:12.188 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:07:12.188 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:07:12.451 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:07:12.451 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:07:17.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:17.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:17.453 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:07:17.492 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:07:17.492 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:07:17.767 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:07:17.767 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:07:22.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:22.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:22.769 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:07:22.809 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:07:22.809 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:07:23.107 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:07:23.724 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:07:28.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:28.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:28.726 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:07:28.768 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:07:28.768 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:07:29.044 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:07:29.044 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:07:34.066 INFO:tasks.rgw_multisite_tests:test notification status upon bucket deletion ... FAIL 2021-02-11T03:07:34.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:34.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:34.067 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:07:34.106 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:07:34.107 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:07:34.344 INFO:teuthology.orchestra.run.smithi057.stderr:No --period given, using current period=702c963a-2caf-4007-8190-7df3d99bc8ac 2021-02-11T03:07:34.386 INFO:teuthology.orchestra.run.smithi057.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1613005980.679832_515.1","last_update":"2021-02-11T01:13:00.679832Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-02-11T03:07:34.386 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-02-11T03:07:34.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:34.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:34.387 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:07:34.429 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:07:34.430 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:07:34.668 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:07:34.668 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-02-11T03:07:34.668 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-02-11T03:07:34.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:34.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:34.669 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:07:34.719 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:07:34.719 INFO:teuthology.orchestra.run.smithi057.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:07:34.998 INFO:teuthology.orchestra.run.smithi057.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613005980.679832_515.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T01:13:00.679832Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:07:34.999 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-02-11T03:07:34.999 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-02-11T03:07:34.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:34.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:34.999 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:07:35.040 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:07:35.040 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:07:35.305 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:07:35.305 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:07:40.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:40.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:40.307 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:07:40.343 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:07:40.344 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:07:40.612 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:07:40.613 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:07:45.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:45.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:45.614 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:07:45.675 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:07:45.676 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:07:45.948 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:07:45.949 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:07:50.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:50.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:50.951 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:07:50.993 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:07:50.993 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:07:51.274 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:07:51.274 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:07:56.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:56.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:07:56.275 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:07:56.323 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:07:56.324 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:07:56.588 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:07:56.588 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:08:01.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:01.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:01.590 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:08:01.630 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:08:01.631 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:08:01.873 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:08:01.874 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:08:06.875 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:07.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:07.516 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:08:07.557 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:08:07.558 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:08:07.820 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:08:07.820 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:08:12.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:12.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:12.822 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:08:12.863 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:08:12.863 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:08:13.143 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:08:13.143 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:08:18.144 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:18.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:18.145 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:08:18.186 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:08:18.186 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:08:18.460 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:08:18.461 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:08:23.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:23.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:23.462 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:08:23.503 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:08:23.503 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:08:23.769 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:08:23.769 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:08:28.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:28.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:28.771 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:08:28.813 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:08:28.813 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:08:29.089 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:08:29.090 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:08:34.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:34.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:34.092 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:08:34.133 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:08:34.133 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:08:34.417 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:08:34.417 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:08:39.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:39.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:39.418 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:08:39.459 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:08:39.459 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:08:39.741 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:08:39.741 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:08:44.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:44.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:44.743 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:08:44.785 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:08:44.785 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:08:45.064 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:08:45.065 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:08:50.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:50.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:50.066 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:08:50.106 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:08:50.106 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:08:50.390 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:08:50.391 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:08:55.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:55.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:08:55.392 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:08:55.433 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:08:55.434 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:08:55.708 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:08:55.708 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:09:00.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:00.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:00.709 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:09:00.750 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:09:00.750 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:09:01.030 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:09:01.030 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:09:06.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:06.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:06.031 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:09:06.072 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:09:06.072 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:09:06.350 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:09:06.350 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:09:11.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:11.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:11.698 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:09:11.739 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:09:11.739 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:09:12.015 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:09:12.556 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:09:17.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:24.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:24.607 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:09:24.733 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:09:24.733 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:09:24.994 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:09:24.995 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:09:29.996 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:29.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:29.997 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:09:30.036 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:09:30.036 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:09:30.302 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:09:30.665 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:09:35.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:35.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:35.667 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:09:35.708 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:09:35.708 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:09:35.980 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:09:35.981 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:09:40.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:40.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:40.982 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:09:41.025 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:09:41.025 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:09:41.298 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:09:41.299 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:09:46.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:46.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:46.300 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:09:46.335 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:09:46.335 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:09:46.585 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:09:46.586 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:09:51.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:51.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:51.587 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:09:51.628 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:09:51.628 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:09:51.902 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:09:51.902 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:09:56.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:56.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:09:56.903 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:09:56.960 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:09:56.961 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:09:57.231 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:09:57.799 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:10:02.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:02.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:02.800 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:10:02.851 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:10:02.851 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:10:03.128 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:10:03.129 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:10:08.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:08.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:08.131 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:10:08.173 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:10:08.173 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:10:08.456 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:10:08.456 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:10:13.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:13.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:13.458 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:10:13.500 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:10:13.500 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:10:13.754 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:10:13.755 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:10:18.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:18.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:18.756 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:10:18.798 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:10:18.799 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:10:19.062 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:10:19.063 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:10:24.064 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:24.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:24.064 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:10:24.104 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:10:24.104 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:10:24.367 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:10:24.367 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:10:29.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:32.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:32.101 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:10:32.139 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:10:32.140 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:10:32.400 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:10:32.400 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:10:37.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:37.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:37.402 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:10:37.507 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:10:37.508 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:10:37.769 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:10:37.770 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:10:42.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:52.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:52.826 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:10:52.869 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:10:52.869 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:10:53.141 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:10:53.425 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:10:58.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:58.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:10:58.426 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:10:58.465 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:10:58.465 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:10:58.732 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:10:58.732 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:11:03.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:11:03.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:11:03.734 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:11:03.776 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:11:03.776 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:11:04.059 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:11:04.059 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:11:09.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:11:09.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:11:09.060 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:11:09.102 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:11:09.102 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:11:09.369 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:11:09.660 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:11:14.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:11:14.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:11:14.661 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:11:14.705 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:11:14.705 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:11:14.979 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:11:14.980 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:11:19.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:11:19.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:11:19.981 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:11:20.024 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:11:20.024 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:11:20.312 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:11:20.313 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:11:25.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:11:25.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:11:25.743 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:11:25.784 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:11:25.784 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:11:26.059 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:11:26.747 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:11:31.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:11:31.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:11:31.749 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:11:31.790 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:11:31.790 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:11:32.093 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:11:32.094 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:11:37.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:11:44.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:11:44.176 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:11:44.216 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:11:44.216 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:11:44.484 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:11:44.484 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:11:49.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:11:49.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:11:49.486 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:11:49.527 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:11:49.527 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:11:49.785 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:11:49.786 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:11:54.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:12:05.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:12:05.003 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:12:05.044 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:12:05.044 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:12:05.304 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:12:05.304 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:12:10.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:12:10.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:12:10.306 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:12:10.338 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:12:10.338 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:12:10.628 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:12:18.704 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:12:23.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:12:24.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:12:24.675 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:12:24.717 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:12:24.717 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:12:24.988 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:12:24.989 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:12:29.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:12:29.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:12:29.990 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:12:30.030 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:12:30.031 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:12:30.317 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:12:30.317 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:12:35.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:12:40.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:12:40.930 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:12:41.045 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:12:41.046 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:12:41.312 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:12:41.313 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:12:46.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:12:46.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:12:46.314 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:12:46.357 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:12:46.357 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:12:46.640 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:12:46.641 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:12:51.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:12:57.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:12:57.651 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:12:57.693 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:12:57.693 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:12:57.961 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:12:57.961 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:13:02.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:13:02.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:13:02.963 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:13:03.004 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:13:03.004 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:13:03.273 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:13:03.273 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:13:08.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:13:17.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:13:17.401 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:13:17.441 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:13:17.441 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:13:17.715 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:13:17.715 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:13:22.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:13:22.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:13:22.716 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:13:22.758 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:13:22.758 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:13:23.030 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:13:33.144 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:13:38.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:13:38.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:13:38.146 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:13:38.187 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:13:38.187 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:13:38.471 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:13:38.472 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:13:43.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:13:43.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:13:43.473 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:13:43.513 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:13:43.513 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:13:43.781 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:13:43.782 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:13:48.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:13:49.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:13:49.991 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:13:50.032 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:13:50.032 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:13:50.307 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:13:50.308 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:13:55.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:13:55.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:13:55.309 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:13:55.352 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:13:55.352 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:13:55.622 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:13:55.622 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:14:00.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:14:08.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:14:08.572 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:14:08.612 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:14:08.612 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:14:08.887 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:14:08.888 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:14:13.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:14:13.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:14:13.889 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:14:13.928 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:14:13.929 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:14:14.217 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:14:14.217 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:14:19.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:14:19.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-02-11T03:14:19.219 DEBUG:teuthology.orchestra.run.smithi095:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-02-11T03:14:19.259 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setuser ceph since I am not root 2021-02-11T03:14:19.260 INFO:teuthology.orchestra.run.smithi095.stderr:ignoring --setgroup ceph since I am not root 2021-02-11T03:14:19.516 INFO:teuthology.orchestra.run.smithi095.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"702c963a-2caf-4007-8190-7df3d99bc8ac","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1613003582.559316_89.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-02-11T00:33:02.559316Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1613003977.863186_285.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-02-11T00:39:37.863186Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-02-11T03:14:19.516 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1613005980.679832_515.1 target=1_1613003977.863186_285.1 2021-02-11T03:14:24.544 INFO:tasks.rgw_multisite_tests:test creating a subscription when no topic info exists ... FAIL 2021-02-11T03:14:25.999 INFO:tasks.rgw_multisite_tests:test updating topic associated with a notification ... ok 2021-02-11T03:14:25.999 INFO:tasks.rgw_multisite_tests:test updating the topic of a notification ... ok 2021-02-11T03:14:26.000 INFO:tasks.rgw_multisite_tests:test notification creation with multiple topics ... ok 2021-02-11T03:14:26.000 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.000 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.000 INFO:tasks.rgw_multisite_tests:FAIL: test low level implementation of s3 notifications 2021-02-11T03:14:26.000 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.000 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.000 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.001 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.001 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 646, in test_ps_s3_notification_low_level 2021-02-11T03:14:26.001 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-02-11T03:14:26.001 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.001 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.001 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.001 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.001 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.001 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.001 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.001 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.001 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.001 INFO:tasks.rgw_multisite_tests:FAIL: test s3 records fetching 2021-02-11T03:14:26.001 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.001 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.002 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.002 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.002 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 719, in test_ps_s3_notification_records 2021-02-11T03:14:26.002 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-02-11T03:14:26.002 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.002 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.002 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.002 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.002 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.002 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.002 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.002 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.002 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.002 INFO:tasks.rgw_multisite_tests:FAIL: test s3 notification set/get/delete 2021-02-11T03:14:26.003 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.003 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.003 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.003 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.003 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 775, in test_ps_s3_notification 2021-02-11T03:14:26.003 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-02-11T03:14:26.003 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.003 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.003 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.003 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.003 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.003 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.003 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.003 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.004 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.004 INFO:tasks.rgw_multisite_tests:FAIL: test s3 topics set/get/delete on master 2021-02-11T03:14:26.004 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.004 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.004 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.004 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.004 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 840, in test_ps_s3_topic_on_master 2021-02-11T03:14:26.004 INFO:tasks.rgw_multisite_tests: master_zone, _ = init_env(require_ps=False) 2021-02-11T03:14:26.004 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.004 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.004 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.004 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.004 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.004 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.005 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.005 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.005 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.005 INFO:tasks.rgw_multisite_tests:FAIL: test s3 topics with secret set/get/delete on master 2021-02-11T03:14:26.005 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.005 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.005 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.005 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.005 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 915, in test_ps_s3_topic_with_secret_on_master 2021-02-11T03:14:26.005 INFO:tasks.rgw_multisite_tests: master_zone, _ = init_env(require_ps=False) 2021-02-11T03:14:26.005 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.005 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.005 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.005 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.006 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.006 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.006 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.006 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.006 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.006 INFO:tasks.rgw_multisite_tests:FAIL: test s3 notification set/get/delete on master 2021-02-11T03:14:26.006 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.006 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.006 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.006 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.006 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 965, in test_ps_s3_notification_on_master 2021-02-11T03:14:26.006 INFO:tasks.rgw_multisite_tests: master_zone, _ = init_env(require_ps=False) 2021-02-11T03:14:26.006 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.006 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.007 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.007 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.007 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.007 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.007 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.007 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.007 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.007 INFO:tasks.rgw_multisite_tests:FAIL: test s3 notification set/get/delete on master 2021-02-11T03:14:26.007 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.007 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.007 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.007 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.007 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 1254, in test_ps_s3_notification_errors_on_master 2021-02-11T03:14:26.007 INFO:tasks.rgw_multisite_tests: master_zone, _ = init_env(require_ps=False) 2021-02-11T03:14:26.008 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.008 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.008 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.008 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.008 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.008 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.008 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.008 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.008 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.008 INFO:tasks.rgw_multisite_tests:FAIL: test set/get/delete of topic 2021-02-11T03:14:26.008 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.008 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.009 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.009 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.009 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 2787, in test_ps_topic 2021-02-11T03:14:26.009 INFO:tasks.rgw_multisite_tests: _, ps_zone = init_env() 2021-02-11T03:14:26.009 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.009 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.009 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.009 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.009 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.009 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.009 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.009 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.009 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.009 INFO:tasks.rgw_multisite_tests:FAIL: test set topic with endpoint 2021-02-11T03:14:26.010 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.010 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.010 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.010 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.010 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 2817, in test_ps_topic_with_endpoint 2021-02-11T03:14:26.010 INFO:tasks.rgw_multisite_tests: _, ps_zone = init_env() 2021-02-11T03:14:26.010 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.010 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.010 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.010 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.010 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.011 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.011 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.011 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.011 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.011 INFO:tasks.rgw_multisite_tests:FAIL: test set/get/delete of notification 2021-02-11T03:14:26.011 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.011 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.011 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.011 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.011 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 2841, in test_ps_notification 2021-02-11T03:14:26.011 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-02-11T03:14:26.011 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.011 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.011 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.012 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.012 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.012 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.012 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.012 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.012 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.012 INFO:tasks.rgw_multisite_tests:FAIL: test set/get/delete of notification on specific events 2021-02-11T03:14:26.012 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.012 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.012 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.012 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.012 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 2879, in test_ps_notification_events 2021-02-11T03:14:26.012 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-02-11T03:14:26.012 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.013 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.013 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.013 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.013 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.013 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.013 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.013 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.013 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.013 INFO:tasks.rgw_multisite_tests:FAIL: test set/get/delete of subscription 2021-02-11T03:14:26.013 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.013 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.013 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.013 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.014 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 2914, in test_ps_subscription 2021-02-11T03:14:26.014 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-02-11T03:14:26.014 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.014 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.014 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.014 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.014 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.014 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.014 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.014 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.014 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.014 INFO:tasks.rgw_multisite_tests:FAIL: test radosgw-admin commands 2021-02-11T03:14:26.014 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.015 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.015 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.015 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.015 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 2986, in test_ps_admin 2021-02-11T03:14:26.015 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-02-11T03:14:26.015 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.015 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.015 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.015 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.015 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.015 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.015 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.016 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.016 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.016 INFO:tasks.rgw_multisite_tests:FAIL: test that events are only sent on incremental sync 2021-02-11T03:14:26.016 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.016 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.016 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.016 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.016 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 3092, in test_ps_incremental_sync 2021-02-11T03:14:26.016 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-02-11T03:14:26.016 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.016 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.016 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.016 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.016 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.017 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.017 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.017 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.017 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.017 INFO:tasks.rgw_multisite_tests:FAIL: test subscriptions for different events 2021-02-11T03:14:26.017 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.017 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.017 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.017 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.017 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 3147, in test_ps_event_type_subscription 2021-02-11T03:14:26.018 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-02-11T03:14:26.018 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.018 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.018 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.018 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.018 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.018 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.018 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.018 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.018 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.018 INFO:tasks.rgw_multisite_tests:FAIL: test incremental fetching of events from a subscription 2021-02-11T03:14:26.018 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.019 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.019 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.019 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.019 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 3286, in test_ps_event_fetching 2021-02-11T03:14:26.019 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-02-11T03:14:26.019 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.019 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.019 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.019 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.019 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.019 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.019 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.019 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.019 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.020 INFO:tasks.rgw_multisite_tests:FAIL: test acking of some events in a subscription 2021-02-11T03:14:26.020 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.020 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.020 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.020 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.020 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 3344, in test_ps_event_acking 2021-02-11T03:14:26.020 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-02-11T03:14:26.020 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.020 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.020 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.020 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.020 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.020 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.021 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.021 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.021 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.021 INFO:tasks.rgw_multisite_tests:FAIL: test object creation notifications in using put/copy/post 2021-02-11T03:14:26.021 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.021 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.021 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.021 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.021 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 3409, in test_ps_creation_triggers 2021-02-11T03:14:26.021 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-02-11T03:14:26.021 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.021 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.021 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.022 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.022 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.022 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.022 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.022 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.022 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.022 INFO:tasks.rgw_multisite_tests:FAIL: test notification of deletion markers 2021-02-11T03:14:26.022 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.022 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.022 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.022 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.022 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 3639, in test_ps_versioned_deletion 2021-02-11T03:14:26.023 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-02-11T03:14:26.023 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.023 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.023 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.023 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.023 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.023 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.023 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.023 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.023 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.023 INFO:tasks.rgw_multisite_tests:FAIL: test notification status upon bucket deletion 2021-02-11T03:14:26.023 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.023 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.023 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.023 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.024 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 4351, in test_ps_delete_bucket 2021-02-11T03:14:26.024 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-02-11T03:14:26.024 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.024 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.024 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.024 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.024 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.024 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.024 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.024 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.024 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-02-11T03:14:26.024 INFO:tasks.rgw_multisite_tests:FAIL: test creating a subscription when no topic info exists 2021-02-11T03:14:26.024 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.025 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-02-11T03:14:26.025 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-02-11T03:14:26.025 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-02-11T03:14:26.025 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 4420, in test_ps_missing_topic 2021-02-11T03:14:26.025 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-02-11T03:14:26.025 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests_ps.py", line 581, in init_env 2021-02-11T03:14:26.025 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-02-11T03:14:26.025 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2021-02-11T03:14:26.025 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-02-11T03:14:26.025 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2021-02-11T03:14:26.025 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-02-11T03:14:26.025 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2021-02-11T03:14:26.025 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.025 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-02-11T03:14:26.026 INFO:tasks.rgw_multisite_tests:Ran 55 tests in 6948.759s 2021-02-11T03:14:26.026 INFO:tasks.rgw_multisite_tests: 2021-02-11T03:14:26.026 INFO:tasks.rgw_multisite_tests:FAILED (failures=21) 2021-02-11T03:14:26.026 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multisite_tests.py", line 74, in begin raise RuntimeError(error_msg + 'test failures') RuntimeError: rgw multisite, rgw multisite pubsub, test failures 2021-02-11T03:14:26.176 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f711f89e09804d03a2fbafc018c7072f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_597c1051df9f309aa7d6c6db46dcbc2c3982c6e3/qa/tasks/rgw_multisite_tests.py", line 74, in begin raise RuntimeError(error_msg + 'test failures') RuntimeError: rgw multisite, rgw multisite pubsub, test failures 2021-02-11T03:14:26.177 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2021-02-11T03:14:26.186 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2021-02-11T03:14:27.952 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2021-02-11T03:14:29.703 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-02-11T03:14:29.704 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T03:14:29.723 INFO:tasks.rgw.c1.client.0.smithi057.stdout:2021-02-11T03:14:29.700+0000 7f49a3fff700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 18677) UID: 0 2021-02-11T03:14:29.723 INFO:tasks.rgw.c1.client.0.smithi057.stdout:2021-02-11T03:14:29.700+0000 7f49c80260c0 -1 shutting down 2021-02-11T03:14:30.326 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-02-11T03:14:35.805 INFO:tasks.rgw.c1.client.0:Stopped 2021-02-11T03:14:43.380 DEBUG:teuthology.orchestra.run.smithi057:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2021-02-11T03:14:43.386 DEBUG:teuthology.orchestra.run.smithi057:> rm -f /home/ubuntu/cephtest/vault-token 2021-02-11T03:14:43.434 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-02-11T03:14:43.434 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T03:14:43.477 INFO:tasks.rgw.c1.client.1.smithi057.stdout:2021-02-11T03:14:43.476+0000 7f6370a64700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 31365) UID: 0 2021-02-11T03:14:43.478 INFO:tasks.rgw.c1.client.1.smithi057.stdout:2021-02-11T03:14:43.476+0000 7f63885bf0c0 -1 shutting down 2021-02-11T03:14:43.482 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2021-02-11T03:14:48.988 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-02-11T03:14:48.988 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-02-11T03:14:49.535 INFO:tasks.rgw.c1.client.1:Stopped 2021-02-11T03:14:49.536 DEBUG:teuthology.orchestra.run.smithi057:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2021-02-11T03:14:49.543 DEBUG:teuthology.orchestra.run.smithi057:> rm -f /home/ubuntu/cephtest/vault-token 2021-02-11T03:14:49.590 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-02-11T03:14:49.591 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T03:14:49.592 INFO:tasks.rgw.c2.client.0.smithi095.stdout:2021-02-11T03:14:49.591+0000 7f6a52234700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 18570) UID: 0 2021-02-11T03:14:49.592 INFO:tasks.rgw.c2.client.0.smithi095.stdout:2021-02-11T03:14:49.591+0000 7f6a69d8f0c0 -1 shutting down 2021-02-11T03:14:54.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-02-11T03:14:59.929 INFO:tasks.rgw.c2.client.0:Stopped 2021-02-11T03:14:59.930 DEBUG:teuthology.orchestra.run.smithi095:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2021-02-11T03:15:00.013 DEBUG:teuthology.orchestra.run.smithi095:> rm -f /home/ubuntu/cephtest/vault-token 2021-02-11T03:15:00.020 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-02-11T03:15:00.020 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T03:15:00.020 INFO:tasks.rgw.c2.client.1:Stopped 2021-02-11T03:15:00.020 DEBUG:teuthology.orchestra.run.smithi095:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2021-02-11T03:15:00.068 DEBUG:teuthology.orchestra.run.smithi095:> rm -f /home/ubuntu/cephtest/vault-token 2021-02-11T03:15:00.116 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2021-02-11T03:15:00.125 INFO:tasks.ceph.ceph_manager.c2:Canceling any pending splits or merges... 2021-02-11T03:15:00.126 DEBUG:teuthology.orchestra.run.smithi095:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-02-11T03:15:00.661 INFO:teuthology.orchestra.run.smithi095.stdout: 2021-02-11T03:15:00.662 INFO:teuthology.orchestra.run.smithi095.stdout:{"epoch":72,"fsid":"20d74388-aeac-4108-8e9a-c22b365ad363","created":"2021-02-11T00:28:46.956056+0000","modified":"2021-02-11T00:43:22.217004+0000","last_up_change":"2021-02-11T00:28:54.536787+0000","last_in_change":"2021-02-11T00:28:54.536787+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T00:28:54.548076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T00:28:55.544852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2021-02-11T00:29:11.217340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2021-02-11T00:29:13.298808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2021-02-11T00:29:27.734902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2021-02-11T00:29:29.033900+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2021-02-11T00:29:31.000994+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2021-02-11T00:29:33.068664+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"test-zone2.rgw.buckets.data","create_time":"2021-02-11T00:29:58.264103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"test-zone2.rgw.log","create_time":"2021-02-11T00:30:00.284364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":11,"pool_name":"test-zone2.rgw.control","create_time":"2021-02-11T00:30:02.054907+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":12,"pool_name":"test-zone2.rgw.meta","create_time":"2021-02-11T00:30:04.128537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":13,"pool_name":"test-zone4.rgw.buckets.data","create_time":"2021-02-11T00:30:33.903953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":14,"pool_name":"test-zone4.rgw.log","create_time":"2021-02-11T00:30:35.267656+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":15,"pool_name":"test-zone4.rgw.control","create_time":"2021-02-11T00:30:37.076875+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":16,"pool_name":"test-zone4.rgw.meta","create_time":"2021-02-11T00:30:39.153951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":17,"pool_name":"test-zone2.rgw.buckets.index","create_time":"2021-02-11T00:32:53.981522+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":18,"pool_name":"test-zone4.rgw.buckets.index","create_time":"2021-02-11T00:32:54.149703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":19,"pool_name":"test-zone2.rgw.otp","create_time":"2021-02-11T00:36:06.333863+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":20,"pool_name":"remove.rgw.log","create_time":"2021-02-11T00:43:14.327601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":21,"pool_name":"remove.rgw.control","create_time":"2021-02-11T00:43:16.233735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":22,"pool_name":"remove.rgw.meta","create_time":"2021-02-11T00:43:18.302116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"e3cce55d-c182-4525-a567-52c86d5dc3db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6809","nonce":14492},{"type":"v1","addr":"172.21.15.95:6811","nonce":14492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6813","nonce":14492},{"type":"v1","addr":"172.21.15.95:6815","nonce":14492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6821","nonce":14492},{"type":"v1","addr":"172.21.15.95:6823","nonce":14492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6818","nonce":14492},{"type":"v1","addr":"172.21.15.95:6820","nonce":14492}]},"public_addr":"172.21.15.95:6811/14492","cluster_addr":"172.21.15.95:6815/14492","heartbeat_back_addr":"172.21.15.95:6823/14492","heartbeat_front_addr":"172.21.15.95:6820/14492","state":["exists","up"]},{"osd":1,"uuid":"618c16cc-ae3f-4158-8f89-2c2e7976780c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6800","nonce":14493},{"type":"v1","addr":"172.21.15.95:6801","nonce":14493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6802","nonce":14493},{"type":"v1","addr":"172.21.15.95:6803","nonce":14493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6806","nonce":14493},{"type":"v1","addr":"172.21.15.95:6807","nonce":14493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6804","nonce":14493},{"type":"v1","addr":"172.21.15.95:6805","nonce":14493}]},"public_addr":"172.21.15.95:6801/14493","cluster_addr":"172.21.15.95:6803/14493","heartbeat_back_addr":"172.21.15.95:6807/14493","heartbeat_front_addr":"172.21.15.95:6805/14493","state":["exists","up"]},{"osd":2,"uuid":"91671609-8e48-43aa-9b1f-f64b54a33d9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6808","nonce":14491},{"type":"v1","addr":"172.21.15.95:6810","nonce":14491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6812","nonce":14491},{"type":"v1","addr":"172.21.15.95:6814","nonce":14491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6819","nonce":14491},{"type":"v1","addr":"172.21.15.95:6822","nonce":14491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.95:6816","nonce":14491},{"type":"v1","addr":"172.21.15.95:6817","nonce":14491}]},"public_addr":"172.21.15.95:6810/14491","cluster_addr":"172.21.15.95:6814/14491","heartbeat_back_addr":"172.21.15.95:6822/14491","heartbeat_front_addr":"172.21.15.95:6817/14491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:52.650890+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:52.509433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:52.755288+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":0,"to":1}]},{"pgid":"3.34","mappings":[{"from":0,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.5","mappings":[{"from":1,"to":0}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":0}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]},{"pgid":"4.23","mappings":[{"from":1,"to":0}]},{"pgid":"4.39","mappings":[{"from":1,"to":2}]},{"pgid":"10.19","mappings":[{"from":1,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T03:15:00.676 DEBUG:teuthology.orchestra.run.smithi095:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2021-02-11T03:15:00.995 INFO:teuthology.misc:Shutting down mds daemons... 2021-02-11T03:15:00.995 INFO:teuthology.misc:Shutting down osd daemons... 2021-02-11T03:15:00.995 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2021-02-11T03:15:00.995 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T03:15:01.082 INFO:tasks.ceph.c2.osd.0:Stopped 2021-02-11T03:15:01.083 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2021-02-11T03:15:01.083 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T03:15:01.172 INFO:tasks.ceph.c2.osd.1:Stopped 2021-02-11T03:15:01.172 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2021-02-11T03:15:01.172 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T03:15:01.262 INFO:tasks.ceph.c2.osd.2:Stopped 2021-02-11T03:15:01.263 INFO:teuthology.misc:Shutting down mgr daemons... 2021-02-11T03:15:01.263 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2021-02-11T03:15:01.263 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T03:15:01.290 INFO:tasks.ceph.c2.mgr.x:Stopped 2021-02-11T03:15:01.290 INFO:teuthology.misc:Shutting down mon daemons... 2021-02-11T03:15:01.290 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2021-02-11T03:15:01.291 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T03:15:01.373 INFO:tasks.ceph.c2.mon.a:Stopped 2021-02-11T03:15:01.374 INFO:tasks.ceph:Checking cluster log for badness... 2021-02-11T03:15:01.374 DEBUG:teuthology.orchestra.run.smithi095:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2021-02-11T03:15:01.456 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi095.front.sepia.ceph.com 2021-02-11T03:15:01.457 DEBUG:teuthology.orchestra.run.smithi095:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2021-02-11T03:15:02.105 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi095.front.sepia.ceph.com 2021-02-11T03:15:02.105 DEBUG:teuthology.orchestra.run.smithi095:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2021-02-11T03:15:02.321 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi095.front.sepia.ceph.com 2021-02-11T03:15:02.321 DEBUG:teuthology.orchestra.run.smithi095:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2021-02-11T03:15:02.497 INFO:tasks.ceph:Archiving mon data... 2021-02-11T03:15:02.499 DEBUG:teuthology.misc:Transferring archived files from smithi095:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi/5874431/data/c2.mon.a.tgz 2021-02-11T03:15:02.499 DEBUG:teuthology.orchestra.run.smithi095:> mktemp 2021-02-11T03:15:02.516 INFO:teuthology.orchestra.run.smithi095.stdout:/tmp/tmp.asR4lWZpys 2021-02-11T03:15:02.516 DEBUG:teuthology.orchestra.run.smithi095:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.asR4lWZpys 2021-02-11T03:15:02.909 DEBUG:teuthology.orchestra.run.smithi095:> sudo chmod 0666 /tmp/tmp.asR4lWZpys 2021-02-11T03:15:02.980 DEBUG:teuthology.orchestra.remote:smithi095:/tmp/tmp.asR4lWZpys is 2MB 2021-02-11T03:15:03.057 DEBUG:teuthology.orchestra.run.smithi095:> rm -fr /tmp/tmp.asR4lWZpys 2021-02-11T03:15:03.064 INFO:tasks.ceph:Cleaning ceph cluster... 2021-02-11T03:15:03.064 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2021-02-11T03:15:03.066 DEBUG:teuthology.orchestra.run.smithi095:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2021-02-11T03:15:03.121 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2021-02-11T03:15:03.130 INFO:tasks.ceph.ceph_manager.c1:Canceling any pending splits or merges... 2021-02-11T03:15:03.130 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-02-11T03:15:03.435 INFO:teuthology.orchestra.run.smithi057.stdout: 2021-02-11T03:15:03.436 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":65,"fsid":"d110339e-4b9e-408b-b214-00afac1a1a23","created":"2021-02-11T00:28:17.792930+0000","modified":"2021-02-11T01:01:40.878664+0000","last_up_change":"2021-02-11T00:28:25.061482+0000","last_in_change":"2021-02-11T00:28:25.061482+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-02-11T00:28:25.079298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-02-11T00:28:26.372533+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2021-02-11T00:29:04.998325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2021-02-11T00:29:07.176221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2021-02-11T00:29:17.084747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2021-02-11T00:29:18.895374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2021-02-11T00:29:20.867758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2021-02-11T00:29:22.962809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"test-zone1.rgw.buckets.data","create_time":"2021-02-11T00:29:40.297139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"test-zone1.rgw.log","create_time":"2021-02-11T00:29:42.036528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":11,"pool_name":"test-zone1.rgw.control","create_time":"2021-02-11T00:29:43.898309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":12,"pool_name":"test-zone1.rgw.meta","create_time":"2021-02-11T00:29:45.942898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":13,"pool_name":"test-zone1.rgw.otp","create_time":"2021-02-11T00:30:15.058373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":14,"pool_name":"test-zone3.rgw.buckets.data","create_time":"2021-02-11T00:30:16.010241+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":15,"pool_name":"test-zone3.rgw.log","create_time":"2021-02-11T00:30:18.078694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":16,"pool_name":"test-zone3.rgw.control","create_time":"2021-02-11T00:30:19.941523+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":17,"pool_name":"test-zone3.rgw.meta","create_time":"2021-02-11T00:30:21.972868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":18,"pool_name":"test-zone1.rgw.buckets.index","create_time":"2021-02-11T00:32:50.776964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":19,"pool_name":"test-zone3.rgw.buckets.index","create_time":"2021-02-11T00:32:54.147092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":20,"pool_name":"test-zone1.rgw.buckets.non-ec","create_time":"2021-02-11T01:01:38.107764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"0e71d1ee-6011-4ad0-8095-04e77220644e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":14536},{"type":"v1","addr":"172.21.15.57:6809","nonce":14536}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":14536},{"type":"v1","addr":"172.21.15.57:6811","nonce":14536}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":14536},{"type":"v1","addr":"172.21.15.57:6815","nonce":14536}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":14536},{"type":"v1","addr":"172.21.15.57:6813","nonce":14536}]},"public_addr":"172.21.15.57:6809/14536","cluster_addr":"172.21.15.57:6811/14536","heartbeat_back_addr":"172.21.15.57:6815/14536","heartbeat_front_addr":"172.21.15.57:6813/14536","state":["exists","up"]},{"osd":1,"uuid":"eecf95ba-73ea-4fc6-963b-011333d6a634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":14537},{"type":"v1","addr":"172.21.15.57:6817","nonce":14537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":14537},{"type":"v1","addr":"172.21.15.57:6819","nonce":14537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":14537},{"type":"v1","addr":"172.21.15.57:6823","nonce":14537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":14537},{"type":"v1","addr":"172.21.15.57:6821","nonce":14537}]},"public_addr":"172.21.15.57:6817/14537","cluster_addr":"172.21.15.57:6819/14537","heartbeat_back_addr":"172.21.15.57:6823/14537","heartbeat_front_addr":"172.21.15.57:6821/14537","state":["exists","up"]},{"osd":2,"uuid":"9a0bccf8-a1ae-4ba3-941d-afda1ae4d2a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":14535},{"type":"v1","addr":"172.21.15.57:6801","nonce":14535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":14535},{"type":"v1","addr":"172.21.15.57:6803","nonce":14535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":14535},{"type":"v1","addr":"172.21.15.57:6807","nonce":14535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":14535},{"type":"v1","addr":"172.21.15.57:6805","nonce":14535}]},"public_addr":"172.21.15.57:6801/14535","cluster_addr":"172.21.15.57:6803/14535","heartbeat_back_addr":"172.21.15.57:6807/14535","heartbeat_front_addr":"172.21.15.57:6805/14535","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:23.603672+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:23.534457+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-11T00:28:22.949429+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":0,"to":2}]},{"pgid":"3.3a","mappings":[{"from":0,"to":1}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]},{"pgid":"4.26","mappings":[{"from":1,"to":0}]},{"pgid":"4.29","mappings":[{"from":1,"to":2}]},{"pgid":"4.3a","mappings":[{"from":1,"to":0}]},{"pgid":"4.3c","mappings":[{"from":1,"to":2}]},{"pgid":"10.10","mappings":[{"from":1,"to":0}]},{"pgid":"14.15","mappings":[{"from":1,"to":2}]},{"pgid":"14.17","mappings":[{"from":1,"to":0}]},{"pgid":"14.22","mappings":[{"from":1,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-11T03:15:03.451 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2021-02-11T03:15:03.775 INFO:teuthology.misc:Shutting down mds daemons... 2021-02-11T03:15:03.775 INFO:teuthology.misc:Shutting down osd daemons... 2021-02-11T03:15:03.776 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2021-02-11T03:15:03.776 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T03:15:03.847 INFO:tasks.ceph.c1.osd.0:Stopped 2021-02-11T03:15:03.848 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2021-02-11T03:15:03.848 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T03:15:03.933 INFO:tasks.ceph.c1.osd.1:Stopped 2021-02-11T03:15:03.933 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2021-02-11T03:15:03.933 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T03:15:03.998 INFO:tasks.ceph.c1.osd.2:Stopped 2021-02-11T03:15:03.998 INFO:teuthology.misc:Shutting down mgr daemons... 2021-02-11T03:15:03.999 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2021-02-11T03:15:03.999 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T03:15:04.023 INFO:tasks.ceph.c1.mgr.x:Stopped 2021-02-11T03:15:04.023 INFO:teuthology.misc:Shutting down mon daemons... 2021-02-11T03:15:04.023 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2021-02-11T03:15:04.023 INFO:teuthology.orchestra.run:waiting for 300 2021-02-11T03:15:04.108 INFO:tasks.ceph.c1.mon.a:Stopped 2021-02-11T03:15:04.108 INFO:tasks.ceph:Checking cluster log for badness... 2021-02-11T03:15:04.108 DEBUG:teuthology.orchestra.run.smithi057:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2021-02-11T03:15:04.124 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi057.front.sepia.ceph.com 2021-02-11T03:15:04.124 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2021-02-11T03:15:04.476 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi057.front.sepia.ceph.com 2021-02-11T03:15:04.476 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2021-02-11T03:15:04.783 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi057.front.sepia.ceph.com 2021-02-11T03:15:04.784 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2021-02-11T03:15:04.891 INFO:tasks.ceph:Archiving mon data... 2021-02-11T03:15:04.891 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi/5874431/data/c1.mon.a.tgz 2021-02-11T03:15:04.891 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2021-02-11T03:15:04.897 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.LlRciwg8Rg 2021-02-11T03:15:04.897 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.LlRciwg8Rg 2021-02-11T03:15:05.300 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.LlRciwg8Rg 2021-02-11T03:15:05.360 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.LlRciwg8Rg is 2MB 2021-02-11T03:15:05.436 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.LlRciwg8Rg 2021-02-11T03:15:05.442 INFO:tasks.ceph:Cleaning ceph cluster... 2021-02-11T03:15:05.442 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2021-02-11T03:15:05.487 DEBUG:teuthology.orchestra.run.smithi095:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2021-02-11T03:15:05.502 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2021-02-11T03:15:05.502 DEBUG:teuthology.orchestra.run.smithi057:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-02-11T03:15:05.546 DEBUG:teuthology.orchestra.run.smithi095:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-02-11T03:15:05.569 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-02-11T03:15:05.675 INFO:teuthology.orchestra.run.smithi095.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-02-11T03:15:05.690 INFO:tasks.ceph:Archiving crash dumps... 2021-02-11T03:15:05.692 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/crash to /home/teuthworker/archive/aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi/5874431/remote/smithi057/crash 2021-02-11T03:15:05.693 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-02-11T03:15:05.714 DEBUG:teuthology.misc:Transferring archived files from smithi095:/var/lib/ceph/crash to /home/teuthworker/archive/aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi/5874431/remote/smithi095/crash 2021-02-11T03:15:05.715 DEBUG:teuthology.orchestra.run.smithi095:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-02-11T03:15:05.735 INFO:tasks.ceph:Compressing logs... 2021-02-11T03:15:05.736 DEBUG:teuthology.orchestra.run.smithi057:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-11T03:15:05.759 DEBUG:teuthology.orchestra.run.smithi095:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-11T03:18:45.463 INFO:tasks.ceph:Archiving logs... 2021-02-11T03:18:45.626 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/log/ceph to /home/teuthworker/archive/aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi/5874431/remote/smithi057/log 2021-02-11T03:18:45.627 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar cz -f - -C /var/log/ceph -- . 2021-02-11T03:19:14.720 DEBUG:teuthology.misc:Transferring archived files from smithi095:/var/log/ceph to /home/teuthworker/archive/aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi/5874431/remote/smithi095/log 2021-02-11T03:19:14.721 DEBUG:teuthology.orchestra.run.smithi095:> sudo tar cz -f - -C /var/log/ceph -- . 2021-02-11T03:19:58.648 DEBUG:teuthology.run_tasks:Unwinding manager install 2021-02-11T03:19:59.430 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2021-02-11T03:19:59.430 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-02-11T03:19:59.433 DEBUG:teuthology.orchestra.run.smithi095:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-02-11T03:19:59.484 DEBUG:teuthology.run_tasks:Unwinding manager clock 2021-02-11T03:19:59.905 INFO:teuthology.task.clock:Checking final clock skew... 2021-02-11T03:19:59.906 DEBUG:teuthology.orchestra.run.smithi057:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-11T03:19:59.932 INFO:teuthology.orchestra.run.smithi057.stdout: remote refid st t when poll reach delay offset jitter 2021-02-11T03:19:59.932 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================== 2021-02-11T03:19:59.932 INFO:teuthology.orchestra.run.smithi057.stdout:-hv01.front.sepi 162.248.241.94 3 u 481 512 377 0.161 1.024 1.178 2021-02-11T03:19:59.932 INFO:teuthology.orchestra.run.smithi057.stdout:*hv02.front.sepi 163.237.218.18 2 u 520 512 377 0.156 -0.105 0.049 2021-02-11T03:19:59.932 INFO:teuthology.orchestra.run.smithi057.stdout:+hv03.front.sepi 74.6.168.72 3 u 494 512 377 0.146 -0.992 0.153 2021-02-11T03:19:59.932 INFO:teuthology.orchestra.run.smithi057.stdout:+hv04.front.sepi 144.172.126.201 3 u 484 512 377 0.143 0.393 0.072 2021-02-11T03:19:59.933 DEBUG:teuthology.orchestra.run.smithi095:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-11T03:20:00.050 INFO:teuthology.orchestra.run.smithi095.stdout: remote refid st t when poll reach delay offset jitter 2021-02-11T03:20:00.051 INFO:teuthology.orchestra.run.smithi095.stdout:============================================================================== 2021-02-11T03:20:00.051 INFO:teuthology.orchestra.run.smithi095.stdout:-hv01.front.sepi 162.248.241.94 3 u 45 512 377 0.176 -0.102 0.298 2021-02-11T03:20:00.051 INFO:teuthology.orchestra.run.smithi095.stdout:*hv02.front.sepi 163.237.218.18 2 u 297 512 377 0.117 -1.682 0.385 2021-02-11T03:20:00.051 INFO:teuthology.orchestra.run.smithi095.stdout:+hv03.front.sepi 216.229.4.69 3 u 47 512 377 0.153 -2.069 0.532 2021-02-11T03:20:00.051 INFO:teuthology.orchestra.run.smithi095.stdout:+hv04.front.sepi 144.172.126.201 3 u 232 512 377 0.137 -0.204 0.945 2021-02-11T03:20:00.051 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-02-11T03:20:00.761 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2021-02-11T03:20:00.761 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2021-02-11T03:20:01.472 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-02-11T03:20:02.981 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2021-02-11T03:20:03.479 INFO:teuthology.task.internal:Duration was 10644.735767 seconds 2021-02-11T03:20:03.480 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-02-11T03:20:03.649 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2021-02-11T03:20:03.650 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-02-11T03:20:03.652 DEBUG:teuthology.orchestra.run.smithi095:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-02-11T03:20:03.765 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2021-02-11T03:20:03.766 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi057.front.sepia.ceph.com 2021-02-11T03:20:03.766 DEBUG:teuthology.orchestra.run.smithi057:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-02-11T03:20:03.778 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi095.front.sepia.ceph.com 2021-02-11T03:20:03.779 DEBUG:teuthology.orchestra.run.smithi095:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-02-11T03:20:03.792 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2021-02-11T03:20:03.792 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-11T03:20:03.822 DEBUG:teuthology.orchestra.run.smithi095:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-11T03:20:03.867 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2021-02-11T03:20:04.274 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2021-02-11T03:20:04.274 DEBUG:teuthology.orchestra.run.smithi057:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-02-11T03:20:04.290 DEBUG:teuthology.orchestra.run.smithi095:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-02-11T03:20:04.314 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2021-02-11T03:20:04.520 DEBUG:teuthology.orchestra.run.smithi057:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-02-11T03:20:04.523 DEBUG:teuthology.orchestra.run.smithi095:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-02-11T03:20:04.535 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = core 2021-02-11T03:20:04.563 INFO:teuthology.orchestra.run.smithi095.stdout:kernel.core_pattern = core 2021-02-11T03:20:04.579 DEBUG:teuthology.orchestra.run.smithi057:> test -e /home/ubuntu/cephtest/archive/coredump 2021-02-11T03:20:04.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T03:20:04.598 DEBUG:teuthology.orchestra.run.smithi095:> test -e /home/ubuntu/cephtest/archive/coredump 2021-02-11T03:20:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-11T03:20:04.628 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-02-11T03:20:05.476 INFO:teuthology.task.internal:Transferring archived files... 2021-02-11T03:20:05.477 DEBUG:teuthology.misc:Transferring archived files from smithi057:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi/5874431/remote/smithi057 2021-02-11T03:20:05.477 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-02-11T03:20:05.501 DEBUG:teuthology.misc:Transferring archived files from smithi095:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/aemerson-2021-02-11_00:10:51-rgw-wip-datalog-genbacking-distro-basic-smithi/5874431/remote/smithi095 2021-02-11T03:20:05.501 DEBUG:teuthology.orchestra.run.smithi095:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-02-11T03:20:05.529 INFO:teuthology.task.internal:Removing archive directory... 2021-02-11T03:20:05.529 DEBUG:teuthology.orchestra.run.smithi057:> rm -rf -- /home/ubuntu/cephtest/archive 2021-02-11T03:20:05.543 DEBUG:teuthology.orchestra.run.smithi095:> rm -rf -- /home/ubuntu/cephtest/archive 2021-02-11T03:20:05.579 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2021-02-11T03:20:06.889 INFO:teuthology.task.internal:Not uploading archives. 2021-02-11T03:20:06.890 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2021-02-11T03:20:07.652 INFO:teuthology.task.internal:Tidying up after the test... 2021-02-11T03:20:07.652 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-11T03:20:07.655 DEBUG:teuthology.orchestra.run.smithi095:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-11T03:20:07.660 INFO:teuthology.orchestra.run.smithi057.stdout: 524339 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 11 03:20 /home/ubuntu/cephtest 2021-02-11T03:20:07.663 INFO:teuthology.orchestra.run.smithi095.stdout: 524339 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 11 03:20 /home/ubuntu/cephtest 2021-02-11T03:20:07.665 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2021-02-11T03:20:12.717 INFO:teuthology.nuke:Checking targets against current locks 2021-02-11T03:20:12.819 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 412, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 269, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 266, in nuke for unnuked in p: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 280, in nuke_one keep_logs = ctx.keep_logs File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/config.py", line 255, in __getattr__ raise AttributeError(name) AttributeError: keep_logs